| 6 Apr 2025 |
 Fernando Rodrigues | Good question. That's what we (and apparently also upstream) need you to explore. Dig into the utility's source code, and try to learn its inner workings. If you find something promising, collect your findings and share them in the issue. | 03:50:44 |
 guider-le-recit | That sounds like fun, thank you Fernando I will start exploring and hopefully get back to you soon. | 03:58:37 |
| 7 Apr 2025 |
| guider-le-recit | Hi Fernando i was wrong this wasn't fun at all, I've analyzed the load_text methods for the B-Tree tables (pinyin_index, phrase_index, etc.) and they seem to process input sequentially without obvious sources of non-deterministic insertion order.
I also traced bigram.db generation to import_interpolation using the Bigram class, confirming it uses DB_HASH. While the Bigram::store method looks deterministic given its SingleGram input, could the non-reproducibility of bigram.db stem from Berkeley DB's default DB_HASH implementation itself being sensitive to the build environment? Is this a known pattern, and are there ways to make BDB Hash generation reproducible? | 14:19:32 |
 raboof | I don't know, but I do remember diffoscope has support for berkely db files, so that is at least a promising sign that people who care about reproducibility have been looking at those :) | 15:20:48 |
| raboof | though "Format-specific differences are supported for Berkeley DB database files but no file-specific differences were detected" in this case so not very helpful :) | 15:37:32 |
| guider-le-recit | The smiley faces help | 15:57:06 |
| guider-le-recit | So instead we are working with BDB's internal metadata, not application level logic right? | 15:57:30 |
| raboof | kinda looks like, yeah | 15:58:12 |
| raboof | seems like there's consistently a difference at 0x34 of the file - might be interesting to see if we can figure out what's written there - I bet some sort of timestamp? - and then where it comes from | 15:59:13 |
| guider-le-recit | Okay thank you, I'll start looking at it, and get back to you | 16:03:29 |
| 8 Apr 2025 |
| guider-le-recit | So I've read into the BDB v9 header format docs. While I couldn't pinpoint the exact field name at 0x34 from the available docs, this area contains generic metadata. I thought thought about what you mentioned about a timestamp but decoding the specific bytes didn't yield obvious Unix timestamps. Wouldn't Another strong candidate for volatile data in that region, based on BDB's design, be LSNs?
I looked for BDB flags to control this. The most promising one seems to be DB_TXN_NOT_DURABLE. Docs explicitly state this removes the LSNs from page headers. Since LSNs/transaction state fit the profile of volatile metadata in that header area potentially sensitive to the environment (ASLR), this flag seems like a direct way to target the suspected cause of the variation. | 12:28:44 |
| guider-le-recit | * So I've read into the BDB v9 header format docs. While I couldn't pinpoint the exact field name at 0x34 from the available docs, this area contains generic metadata. I thought about what you mentioned about a timestamp but decoding the specific bytes didn't yield obvious Unix timestamps. Wouldn't Another strong candidate for volatile data in that region, based on BDB's design, be LSNs?
I looked for BDB flags to control this. The most promising one seems to be DB_TXN_NOT_DURABLE. Docs explicitly state this removes the LSNs from page headers. Since LSNs/transaction state fit the profile of volatile metadata in that header area potentially sensitive to the environment (ASLR), this flag seems like a direct way to target the suspected cause of the variation. | 12:29:07 |
| guider-le-recit | But im not entirely sure on this though | 12:30:16 |
| raboof | nice research work! this is deeper into BDB internals than I've gotten, but it sounds worth a try? | 12:41:48 |
| 9 Apr 2025 |
|  gigamonster256 joined the room. | 17:04:16 |
| 10 Apr 2025 |
| raboof | generated another report for the minimal iso, no surprises (gettext and jemalloc are still in the staging pipeline) https://reproducibility.nixos.social/reports/nixos-minimal-25.05pre780821.c8cd81426f45-x86_64-linux.iso | 06:44:57 |
| 11 Apr 2025 |
|  Alexander (Axler1) joined the room. | 12:32:19 |
 3pleX-dev | 
Download ima_2f2583a.png | 14:00:01 |
| 3pleX-dev | Hi Fernando Rodrigues: please I’d like to ask if you have a timeline set for the reproducible build project? It’s states here that we should work with our mentors to provide a timeline for the work | 14:00:11 |
| Fernando Rodrigues | In reply to @3plexdev:matrix.org
Hi Fernando Rodrigues: please I’d like to ask if you have a timeline set for the reproducible build project? It’s states here that we should work with our mentors to provide a timeline for the work This is a bit difficult to answer at this point because we aren't yet sure if we'll be able to assign the intern with a certain special project, and because each r13y issue varies wildly in complexity and scope. For now, check the calendar and simply tell Outreachy that the internship begins in the second day of June and has a second part beginning July 15. You'll be assigned various issues and are expected to research the inner workings of third-party applications for 30h/week to hopefully fix a number of unreproducible packages. | 14:54:15 |
| 3pleX-dev | In reply to @sigmasquadron:matrix.org
This is a bit difficult to answer at this point because we aren't yet sure if we'll be able to assign the intern with a certain special project, and because each r13y issue varies wildly in complexity and scope. For now, check the calendar and simply tell Outreachy that the internship begins in the second day of June and has a second part beginning July 15. You'll be assigned various issues and are expected to research the inner workings of third-party applications for 30h/week to hopefully fix a number of unreproducible packages. Ah okay! Thank you so much — this was really helpful | 15:35:54 |
 emily | https://fedoraproject.org/wiki/Changes/Package_builds_are_expected_to_be_reproducible | 18:59:12 |
| emily | hey we get a shoutout in here even :) | 18:59:13 |
| emily | https://lwn.net/Articles/1014979/ LWN article | 18:59:25 |
|  @ambroisie:belanyi.fr left the room. | 22:07:46 |
| 12 Apr 2025 |
|  @loving-melody:matrix.org left the room. | 02:17:17 |
|  Philip changed their profile picture. | 11:36:07 |
| guider-le-recit | Apologies for the delayed update on this, in essence I modified the libpinyin source (ngram_bdb.cpp, chewing_large_table2_bdb.cpp, phrase_large_table3_bdb.cpp, punct_table_bdb.cpp) to add a DB set_flags(handle, DB_TXN_NOT_DURABLE) call immediately after db_create() and before DB open() for all database handles used during index generation.
However, attempting to build failed during the make process when running gen_binary_files. The build log showed: ''BDB1566 DB_NOT_DURABLE interface requires an environment configured for the transaction subsystem''
Given that DB_TXN_NOT_DURABLE did not work, I reverted those changes and went back to test the difference between access methods. I modified only ngram_bdb.cpp to change the bigram.db file type from DB_HASH to DB_BTREE in all its DB open calls.
The build completed this time, but the reproducibility check (--check) still failed. Running diffoscope showed that while bigram.db is now reported as a B-Tree file, it exhibits the exact same header difference pattern around offset 0x34 as all the other B-Tree files
I did not want to try refractoring a DB ENV, so I tried looking if there where any flags i missed that could handle this, there aren't any but instead I found that the problematic region starting at 0x34 corresponds to a 20-byte uid field within the common DBMETA structure
(https://github.com/zvelo/BerkeleyDB/blob/master/src/dbinc/db_page.h)
This uid is an apperantly inherently non-deterministic unique file identifier generated by BDB during database creation, influenced by runtime factors potentially including ASLR. BDB docs confirms there are no API flags controllable via DB->set_flags() on standalone handles to suppress or stabilize this uid generation.
I guess it seems that, the non-reproducibility affecting all generated BDB files stem directly from this volatile uid field. At this point i am tired and not sure what to do next | 20:05:15 |
| Fernando Rodrigues | In reply to @guider-le-recit:matrix.org
Apologies for the delayed update on this, in essence I modified the libpinyin source (ngram_bdb.cpp, chewing_large_table2_bdb.cpp, phrase_large_table3_bdb.cpp, punct_table_bdb.cpp) to add a DB set_flags(handle, DB_TXN_NOT_DURABLE) call immediately after db_create() and before DB open() for all database handles used during index generation.
However, attempting to build failed during the make process when running gen_binary_files. The build log showed: ''BDB1566 DB_NOT_DURABLE interface requires an environment configured for the transaction subsystem''
Given that DB_TXN_NOT_DURABLE did not work, I reverted those changes and went back to test the difference between access methods. I modified only ngram_bdb.cpp to change the bigram.db file type from DB_HASH to DB_BTREE in all its DB open calls.
The build completed this time, but the reproducibility check (--check) still failed. Running diffoscope showed that while bigram.db is now reported as a B-Tree file, it exhibits the exact same header difference pattern around offset 0x34 as all the other B-Tree files
I did not want to try refractoring a DB ENV, so I tried looking if there where any flags i missed that could handle this, there aren't any but instead I found that the problematic region starting at 0x34 corresponds to a 20-byte uid field within the common DBMETA structure
(https://github.com/zvelo/BerkeleyDB/blob/master/src/dbinc/db_page.h)
This uid is an apperantly inherently non-deterministic unique file identifier generated by BDB during database creation, influenced by runtime factors potentially including ASLR. BDB docs confirms there are no API flags controllable via DB->set_flags() on standalone handles to suppress or stabilize this uid generation.
I guess it seems that, the non-reproducibility affecting all generated BDB files stem directly from this volatile uid field. At this point i am tired and not sure what to do next This is great! The next step would be to iteratively implement a deterministic way to generate the unique file identifier, which would mean patching BerkeleyDB.
But we don't need to worry about that right now. Since the deadline for outreachy applications is coming up, please merge all of your research and your messages here in a GitHub issue, so you have a link to post on Outreachy. | 20:10:08 |
