| 25 May 2021 |
 andi- | In reply to @raboof:matrix.org
andi-: I'm not sure, but I think we didn't figure out how to reproduce them reliably. In https://github.com/NixOS/nixpkgs/pull/120858 we didn't manage to reproduce them anymore either. but we still don't allow optimized builds when the reproducible flage is on :/ | 14:02:17 |
| andi- | My concern is primarily the optimisations for the interpreter and not so much the bytecode.
I did just run the latest master version against a Debian python 3.8 build and the difference is for some tests significant.
Nothing that warrants giving up reproducibility tho..
+-------------------------+--------------+-----------------+--------------+------------------------+
| Benchmark | nixpy38.json | py38docker.json | Change | Significance |
+=========================+==============+=================+==============+========================+
| 2to3 | 598 ms | 438 ms | 1.37x faster | Significant (t=76.86) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| chaos | 197 ms | 166 ms | 1.19x faster | Significant (t=44.94) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| deltablue | 12.2 ms | 10.7 ms | 1.14x faster | Significant (t=25.92) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| fannkuch | 627 ms | 688 ms | 1.10x slower | Significant (t=-30.21) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| float | 179 ms | 162 ms | 1.11x faster | Significant (t=23.09) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| go | 399 ms | 375 ms | 1.06x faster | Significant (t=19.53) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| hexiom | 16.1 ms | 14.3 ms | 1.13x faster | Significant (t=22.71) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| json_dumps | 17.7 ms | 16.1 ms | 1.10x faster | Significant (t=20.39) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| json_loads | 38.1 us | 30.9 us | 1.23x faster | Significant (t=37.52) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| logging_format | 15.0 us | 13.4 us | 1.11x faster | Significant (t=16.43) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| logging_silent | 322 ns | 285 ns | 1.13x faster | Significant (t=19.30) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| logging_simple | 13.8 us | 12.3 us | 1.12x faster | Significant (t=17.29) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| meteor_contest | 132 ms | 132 ms | 1.00x faster | Not significant |
+-------------------------+--------------+-----------------+--------------+------------------------+
| nbody | 200 ms | 201 ms | 1.01x slower | Not significant |
+-------------------------+--------------+-----------------+--------------+------------------------+
| nqueens | 164 ms | 146 ms | 1.13x faster | Significant (t=30.04) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| pathlib | 28.6 ms | 25.6 ms | 1.12x faster | Significant (t=22.13) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| pickle | 13.7 us | 11.8 us | 1.16x faster | Significant (t=31.15) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| pickle_dict | 32.8 us | 24.6 us | 1.34x faster | Significant (t=41.78) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| pickle_list | 4.30 us | 3.97 us | 1.08x faster | Significant (t=17.17) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| pickle_pure_python | 790 us | 702 us | 1.12x faster | Significant (t=42.31) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| pidigits | 181 ms | 183 ms | 1.01x slower | Not significant |
+-------------------------+--------------+-----------------+--------------+------------------------+
| pyflate | 1.03 sec | 1.00 sec | 1.03x faster | Significant (t=10.36) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| python_startup | 28.1 ms | 9.23 ms | 3.05x faster | Significant (t=365.10) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| python_startup_no_site | 14.1 ms | 6.19 ms | 2.28x faster | Significant (t=252.71) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| raytrace | 891 ms | 778 ms | 1.14x faster | Significant (t=44.80) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| regex_compile | 272 ms | 252 ms | 1.08x faster | Significant (t=19.34) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| regex_dna | 183 ms | 212 ms | 1.16x slower | Significant (t=-45.28) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| regex_effbot | 3.44 ms | 3.42 ms | 1.01x faster | Not significant |
+-------------------------+--------------+-----------------+--------------+------------------------+
| regex_v8 | 29.5 ms | 27.7 ms | 1.07x faster | Significant (t=21.86) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| richards | 119 ms | 113 ms | 1.06x faster | Significant (t=11.34) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| scimark_fft | 580 ms | 527 ms | 1.10x faster | Significant (t=29.30) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| scimark_lu | 264 ms | 234 ms | 1.13x faster | Significant (t=19.78) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| scimark_monte_carlo | 179 ms | 151 ms | 1.19x faster | Significant (t=30.89) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| scimark_sor | 326 ms | 292 ms | 1.11x faster | Significant (t=25.12) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| scimark_sparse_mat_mult | 8.19 ms | 7.41 ms | 1.10x faster | Significant (t=12.68) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| spectral_norm | 238 ms | 205 ms | 1.16x faster | Significant (t=36.94) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| sqlite_synth | 4.98 us | 4.28 us | 1.16x faster | Significant (t=46.00) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| telco | 11.2 ms | 8.06 ms | 1.39x faster | Significant (t=68.59) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| unpack_sequence | 53.9 ns | 134 ns | 2.48x slower | Significant (t=-55.83) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| unpickle | 23.1 us | 18.5 us | 1.25x faster | Significant (t=42.93) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| unpickle_list | 5.15 us | 6.09 us | 1.18x slower | Significant (t=-33.21) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| unpickle_pure_python | 557 us | 497 us | 1.12x faster | Significant (t=22.79) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| xml_etree_generate | 167 ms | 137 ms | 1.22x faster | Significant (t=28.84) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| xml_etree_iterparse | 135 ms | 129 ms | 1.04x faster | Significant (t=6.33) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| xml_etree_parse | 191 ms | 177 ms | 1.08x faster | Significant (t=18.17) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| xml_etree_process | 130 ms | 108 ms | 1.20x faster | Significant (t=67.33) |
+-------------------------+--------------+-----------------+--------------+------------------------+
| 14:04:09 |
 raboof | In reply to @andi:kack.it
but we still don't allow optimized builds when the reproducible flage is on :/ do you mean on master or on that PR? | 14:06:14 |
| andi- | Both. That PR you linked is only concerned about bytecode. | 14:07:35 |
| andi- | I'm trying to build the python3Minimal package now with optimisations and that assert disabled to see why it isn't reproducible. | 14:10:06 |
| raboof | ah yes. I keep conflating 'enabling optimizations' (--enable-optimizations) and shipping optimized bytecode (-OO) in my head, but those are separate things, right? | 14:11:23 |
| raboof | I'm not much of a pythonista ;) | 14:12:22 |
|  philipp left the room. | 14:13:05 |
| andi- | Oh joy, touching pythonMinimal is an stdenv rebuild :D Time to introduce a temporary test attribute.. | 14:19:03 |
 hexa | it would probably be prudent to … well … invite FRidh | 14:20:54 |
| andi- | Trust me I would have if I could... | 14:23:37 |
| hexa | poked over in #python:nixos.org | 14:31:07 |
|  nbathum (he or they) changed their display name from nbathum to nbathum (he or they). | 14:36:26 |
| nbathum (he or they) set a profile picture. | 14:36:30 |
| andi- | In reply to @hexa:lossy.network
poked over in #python:nixos.org I did already send him a DM about this... but thanks for getting involved... | 14:40:55 |
|  atemu12 joined the room. | 15:46:00 |
| andi- | I might have some patches that are similar effective as the --enable-optimisations hammer but aren't hindering reproducibility. The first one is good. Trying the second patch just now but I have to rebootstrap everything for that... | 18:07:26 |
 @grahamc:nixos.org | is there something I can do to help? | 18:58:14 |
| andi- | Not sure :D | 19:02:57 |
| andi- | I just finished the 2nd commit. Will open an PR now and then there shall be discussion! | 19:03:09 |
| andi- | https://github.com/NixOS/nixpkgs/pull/124416 | 19:14:07 |
| andi- | I might have found one of the sources of PGO builds being non-deterministic: https://github.com/miss-islington/cpython/blob/16270bf2f945eea4c430000d372c30c877523b20/Lib/test/libregrtest/main.py#L259 | 19:42:52 |
| andi- | There is probably still the problem of impure compute happening within the builds (e.g. current time being used for string formatting etc..) | 19:44:03 |
| andi- | mhm that defaults to non-random execution... It could have been so simple :-) | 19:46:07 |
| andi- | Quality hardware. My M1 Mac that I am testing the darwin changes on just restarted. This is the 2nd time in 2 weeks that it dies during an stdenv rebuild.... | 20:30:45 |
| andi- | I am giving up on that architecture... The machine crashes on a single job with 4 cores during the bootstrap... shrug | 20:49:16 |
 baloo | andi: afaict, python was reproducible on >3.6 | 23:07:28 |
| baloo | unless it changed recently | 23:07:40 |
| andi- | In reply to @baloo_:matrix.org
andi: afaict, python was reproducible on >3.6 Well I changed the compile flags in the linked PR ;) | 23:08:15 |
| baloo | pretty sweet | 23:09:14 |
