lzma module parity issues between platforms

[Fabizocker456]

Bug report

Bug description:

file = bytes.fromhex("5d 00 00 00 01 1c 03 00 00 00 00 00 00 00 22 9e 0a 07 11 8f 47 b1 b9 0d 6f 94 20 29 e9 bd 75 36 2c fa d5 b3 21 74 52 ae e3 20 73 96 e5 c9 5c 38 fa a8 e5 74 22 47 9d 41 e1 6e 0b c3 62 25 4c c7 24 35 d6 10 4c ce db 51 a1 da 26 29 42 8d b2 1d 54 56 b2 5f 51 10 b8 bd e0 5a 6a 29 dd 95 63 b9 f4 bc 79 37 50 ed 61 2b ec 2c 8b 25 96 40 30 5b f5 f2 dc 28 f7 75 3b 7b dd cf 7a 81 8a af 81 81 aa 74 bb 01 e3 df 2d 84 7b 5e 84 38 89 c9 c3 ac cf 52 f2 11 71 89 55 58 61 43 1f a2 7e 39 e7 30 74 01 35 a8 1f 40 f7 d7 15 6e 32 b3 cb 0d 7e 71 07 dc 76 4f 51 af 72 f5 5f e4 3d 54 82 3e 11 29 b0 01 b9 ea a7 fb fe db 17 4b b3 6f 2b 95 0e 22 e2 ea 86 72 d0 0c 5b cb de bd 78 92 20 07 96 e0 8c 8b 7d 71 8d 41 d9 cd 1e 22 fb 35 b3 b8 72 a1 b3 a0 ee 60 d3 04 1b 53 f3 57 62 23 6d 47 3e 7d 49 81 7f a2 67 98 7d cc e8 19 d0 93 4c 43 23 60 27 ea 80 e3 bf 49 5b c6 19 63 63 87 54 d9 61 67 0c 14 bb bf 84 bb 31 55 d8 de d9 2d 48 30 b8 c8 9b 71 eb 0c 74 98 a2 93 23 f7 a9 29 a0 25 27 58 e6 48 fd db 77 ab e8 67 cf 55 41 ec f2 36 52 5d 87 60 ba fa 71 f3 e5 fc cd 8c 02 53 b8 bf cf e3 59 97 53 36 37 f5 7c 1b cd e9 8e 0e 59 7a 33 6b 32 63 f6 3f f7 25 fc 94 8f f3 be 1f dd 5d 9d c5 82 10 d9 a0 ac f2 cc b8 1c 09 b6 16 4c e8 36 61 ef a3 48 94 9f fa 42 ed 88 66 d8 4f da 8a fa 4a ff 15 c0 26 88 45 db ff f3 a1 34 cd f9 69 e5 cf 27 0d 6e 1b 65 2a c3 f9 4b 85 f7 3c 45 84 30 af 80 e9 38 34 dc 83 d7 b9 54 79 ea d6 17 3c 22 97 68 4e 72 fa d2 e7 03 b9 33 fd b1 ee 7c 79 48 01 0a 57 33 29 d1 2c fa 5b ff ff e1 40 2d dc")

import lzma
lzma.decompress(file)

The above code does not work on a fresh Python 3.12 install under Windows 10 and 11 (_lzma.LZMAError: Corrupt input data), it does however work on a self-built Python 3.12 on Debian Linux, and probably Linux in general. (I, myself, have tested on Win11, Win10 and self-built Debian, a few other users have on unspecified Linuxes and Windowses).

This difference between platforms is, as far as I know, not intentional (and if it is, this is not mentioned in the lzma module documentation).

(the raw data is sourced from https://origin.warframe.com/PublicExport/index_en.txt.lzma, (C) Digital Extremes, probably)

CPython versions tested on:

3.12

Operating systems tested on:

Linux, Windows

[picnixz]

I can confirm that on POSIX, we get something like:

b'ExportCustoms_en.json!00_wzwkOFB7XPlst9tXiSU55g\r\nExportDrones_en.json!00_LZKVJWzyNlEXy-hI3mpGnA\r\nExportFlavour_en.json!00_7CEYAqJB7CoydL67fjhBnQ\r\nExportFusionBundles_en.json!00_yOD81zJOvs6qph81j6McIQ\r\nExportGear_en.json!00_9c0bmrHcYYWi+mecnxaL-A\r\nExportKeys_en.json!00_Lp76eoKm5m7LiXCvzbwQ2Q\r\nExportRecipes_en.json!00_iVEngekAzDO6rhc3Sva+gQ\r\nExportRegions_en.json!00_Hs3LJXyk-r-QQ0YV-GgwpA\r\nExportRelicArcane_en.json!00_ElWYUJFXc9WokdrB08GxCg\r\nExportResources_en.json!00_2fIZAJKmcZEpsIkFE8kMiw\r\nExportSentinels_en.json!00_kWACswtGMTzojXnsLlUavQ\r\nExportSortieRewards_en.json!00_+ULn-PCN7yAvvr2M0mPUIg\r\nExportUpgrades_en.json!00_2T76qHmKfdDy2STiCEsgVg\r\nExportWarframes_en.json!00_xk-QTfzHI6Ijy0WkyCcYEQ\r\nExportWeapons_en.json!00_mrFPI-tnYdaXwZkA8dRNaA\r\nExportManifest.json!00_sIif74qNmZa-XrLaoVgWnw'

By the way, I tested it on online interpreters and the POSIX ones running Python 3.8 also had the same error.

[Fabizocker456]

I can confirm that on POSIX, we get something like:

b'ExportCustoms_en.json!00_wzwkOFB7XPlst9tXiSU55g\r\nExportDrones_en.json!00_LZKVJWzyNlEXy-hI3mpGnA\r\nExportFlavour_en.json!00_7CEYAqJB7CoydL67fjhBnQ\r\nExportFusionBundles_en.json!00_yOD81zJOvs6qph81j6McIQ\r\nExportGear_en.json!00_9c0bmrHcYYWi+mecnxaL-A\r\nExportKeys_en.json!00_Lp76eoKm5m7LiXCvzbwQ2Q\r\nExportRecipes_en.json!00_iVEngekAzDO6rhc3Sva+gQ\r\nExportRegions_en.json!00_Hs3LJXyk-r-QQ0YV-GgwpA\r\nExportRelicArcane_en.json!00_ElWYUJFXc9WokdrB08GxCg\r\nExportResources_en.json!00_2fIZAJKmcZEpsIkFE8kMiw\r\nExportSentinels_en.json!00_kWACswtGMTzojXnsLlUavQ\r\nExportSortieRewards_en.json!00_+ULn-PCN7yAvvr2M0mPUIg\r\nExportUpgrades_en.json!00_2T76qHmKfdDy2STiCEsgVg\r\nExportWarframes_en.json!00_xk-QTfzHI6Ijy0WkyCcYEQ\r\nExportWeapons_en.json!00_mrFPI-tnYdaXwZkA8dRNaA\r\nExportManifest.json!00_sIif74qNmZa-XrLaoVgWnw'

This is the same result I got, and probably the correct result. The PeaZip tool gives the same result under Windows, too.

[zooba]

Can anyone confirm xz versions? We don't implement this ourselves, we rely on other libraries - typically the ones bundled in an OS for Linux.