Extremely slow when using specific commands

[bdnbje]

Using the dict_size (lzma) method in mkdwarfs slows the encoding process exponentially which is not the ideal for this command. Using the same in 7z does it much quicker. This could because of the liblzma API from XZ utils is not ideal. lzip's or Igor's(get's updated way often is ideal.

also the lzma format describes way more finer controls like lc,lp,pb,mf,mc,fb which can shave a lot of bits since the methods described as level 9 is not ideal for a lot of files.

[mhx]

I'll turn this into a discussion for now as this is a bit speculative and it's not completely clear to me what would be the best way forward.

[bdnbje]

can we get to set the lc,lp,pb,mf,mc,fb settings?

[mhx]

This could because of the liblzma API from XZ utils is not ideal. lzip's or Igor's(get's updated way often is ideal.

Sadly, xz-utils is what distros bundle. I'm not aware of distros bundling the 7-zip SDK.

can we get to set the lc,lp,pb,mf,mc,fb settings?

Mostly. mc and fb don't seem to be exposed through the API. Maybe they were in the past and have been removed now?

[mhx]

@bdnbje, can you play around with the lzma-options branch, which now has support for the following:

  lzma     LZMA compression
               level=[0..9]
               dict_size=[12..30]
               extreme
               binary={x86,powerpc,ia64,arm,armthumb,sparc}
               lc=[0..4]
               lp=[0..4]
               pb=[0..4]
               mf={hc3,hc4,bt2,bt3,bt4}
               mode={fast,normal}
               nice=[2..273]
               depth=[4..1000]

[mhx]

Using the dict_size (lzma) method in mkdwarfs slows the encoding process exponentially which is not the ideal for this command. Using the same in 7z does it much quicker.

Can you provide a real example of the slowdown you're seeing?

I've just quickly tried to compare the two and here's what I get:

$ mkdir test
$ dd if=/dev/urandom of=test/random.data bs=1M count=64
$ time ./mkdwarfs -i test -o test.dwarfs -S 26 -C lzma:dict_size=26
23.72s user 0.26s system 99% cpu 23.977 total
$ time ./7zip/7zzs -mmt1 -md=64m a test.7z test/random.data
19.71s user 0.26s system 99% cpu 20.049 total

That's with the latest 7z release (22.00).

[bdnbje]

I did some testing on a mame romset. level 9 with the extreme setting beats out any of the new settings.
./mkdwarfs -i Downloads/ -o mame7.dwarfs --order path -N 16 -L 32G -S 26 -B 4 -W 12 -C lzma:level=9:extreme
1459733131 mame7.dwarfs
./mkdwarfs -i Downloads/ -o mame6.dwarfs --order path -N 16 -L 32G -S 26 -B 4 -W 12 -C lzma:dict_size=28:lc=4:lp=0:pb=3:mf=bt4:mode=normal:nice=273:depth=10000
1461794271 mame6.dwarfs
I will try it on other data sets.
Igor's 7-zip sdk could be shipped with the dwarfs code can't it?
Dwarfs is aimed to be a high performance fs and 7zip sdk might give it some performance increments as it gets maintained much often than the xz-utils.

[mhx]

1459733131 mame7.dwarfs
1461794271 mame6.dwarfs

Besides, that's a 0.14% difference.

Igor's 7-zip sdk could be shipped with the dwarfs code can't it?

It certainly could, but I'd rather not bundle any more stuff with DwarFS. There are already requests to "unbundle" libraries from the codebase. If the SDK version is in fact much better than the xz version, I'm wondering why there are no packages for most/all distributions given the code is in the public domain?

In any case, if you want to play with this further, I've added a small Python script that allows you to extract all blocks from a DwarFS image. Then you can experiment with compressing these blocks individually (that's essentially what DwarFS does internally) and see if you can get any significant improvement. To do this, first create an image and disable block compression completely:

mkdwarfs -i /usr/include -o test.dwarfs -l0 -S26 --order=nilsimsa -W12 -w4

Alternatively, you can use an existing DwarFS image and create an uncompressed copy of it:

mkdwarfs -i compressed.dwarfs -o test.dwarfs --recompress -C none --schema-compression=none --metadata-compression=none

Then, extract the blocks:

$ scripts/extract_blocks.py test.dwarfs include_
DWARFS v2.4 [0] block (none) 67108864 bytes
DWARFS v2.4 [1] block (none) 67108864 bytes
DWARFS v2.4 [2] block (none) 67108864 bytes
DWARFS v2.4 [3] block (none) 67108864 bytes
DWARFS v2.4 [4] block (none) 67108864 bytes
DWARFS v2.4 [5] block (none) 67108864 bytes
DWARFS v2.4 [6] block (none) 45780855 bytes
DWARFS v2.4 [7] schema (none) 1136 bytes
DWARFS v2.4 [8] metadata (none) 1615300 bytes
DWARFS v2.4 [9] index (none) 80 bytes

$ ll include_*
.rw-r--r-- 67108864 mhx users  4 Jul 12:54 include_block0
.rw-r--r-- 67108864 mhx users  4 Jul 12:54 include_block1
.rw-r--r-- 67108864 mhx users  4 Jul 12:54 include_block2
.rw-r--r-- 67108864 mhx users  4 Jul 12:54 include_block3
.rw-r--r-- 67108864 mhx users  4 Jul 12:54 include_block4
.rw-r--r-- 67108864 mhx users  4 Jul 12:54 include_block5
.rw-r--r-- 45780855 mhx users  4 Jul 12:54 include_block6
.rw-r--r--       80 mhx users  4 Jul 12:54 include_index9
.rw-r--r--  1615300 mhx users  4 Jul 12:54 include_metadata8
.rw-r--r--     1136 mhx users  4 Jul 12:54 include_schema7

[Phantop]

If compression speeds with LZMA are an issue, maybe consider FLZMA? There's a fork of XZ utils that uses it too so it might be easier to get it.

https://github.com/conor42/fast-lzma2
https://github.com/conor42/fxz

[bdnbje]

One reason 7zip sdk is faster could be due to it's code written in ASM. it has x86 asm and arm asm support too. XZ is a derivative of this sdk and what i could find is it got popular due to it being functionally similar to the zlib library. also xz is in cpp. I will definitely try to the new python script and find more about it thanks.

[fwn0]

I am not compression guru, but I see that my every packed apps with 7z are smallest than dwarfs (brotli:quality=11:lgwin=28 or zstd:level=21). Here is one of much other examples - cromite browser:

chrome-lin unpacked: 790,7 mb
github package tar.gz: 225,4 mb
wrappe zstd binary: 165,9 mb
dwarfs zstd: 157,0 mb
dwarfs brotli: 147,8 mb
7zzs ultra tuned: 135,3 mb

I understood problem with an available library for 7z, but I use 7zzs (static binary); is it not possible to insert this binary into dwarfs or some hacking for linking to this binary in your compression algorithm?

[fwn0]

I have created new dwarfs from the folder with 7zzs and cromite.7z - 136.6 mb (cromite7z.dwarfs). How can I create a header script to extract 7z and launch dwarfs?

[mhx]

You can't quite compare the compression ratio of DwarFS images with that of archive formats such as 7z. DwarFS is a file system, so the compressed data is optimized for random access. Archive formats are usually optimized purely for compression ratio.

Nevertheless, as you suggested, I've done a comparison using chromium 133.0.6894.1:

$ du -hs chromium-133.0.6894.1
4.9G	chromium-133.0.6894.1

$ time 7zz a -mx9 chromium-133.0.6894.1.7z chromium-133.0.6894.1
[...]
CPU	878%
user	19:52.77
system	8.851
total	2:16.85

$ time mkdwarfs -i chromium-133.0.6894.1 -o chromium-133.0.6894.1.dwarfs -l9 -L8g
[...]
CPU	1637%
user	25:33.33
system	56.975
total	1:37.11

$ ll chromium-133.0.6894.1.*
.rw-r--r-- 1,062,674,488 mhx mhx 13 Dec 22:17 chromium-133.0.6894.1.7z
.rw-r--r-- 1,065,982,330 mhx mhx 13 Dec 22:19 chromium-133.0.6894.1.dwarfs

So, as you can see, without much tweaking (the -L8g is just to make better use of the CPU resources), the DwarFS image is just 0.3% larger than the 7z archive. mkdwarfs uses slightly more CPU than 7zz, but it finishes much faster in terms of wallclock speed.

[fwn0]

hmmm ... let see my 7z achive:

./7zzs a -t7z -snl -mx=9 -mfb=273 -ms -md=31 -myx=9 -mtm=- -mmt -mmtf -md=2536m -mmf=bt3 -mmc=10000 -mpb=0 -mlc=0 chromium-133.0.6894.1.7z chromium-133.0.6894.1

7-Zip (z) 24.09 (x64) : Copyright (c) 1999-2024 Igor Pavlov : 2024-11-28
64-bit locale=en_US.UTF-8 Threads:4 OPEN_MAX:1024, ASM

Scanning the drive:
40633 folders, 456475 files, 3929609701 bytes (3748 MiB)

Creating archive: chromium-133.0.6894.1.7z

Add new data to archive: 40633 folders, 456475 files, 3929609886 bytes (3748 MiB)

Files read from disk: 455666
Archive size: 961855808 bytes (918 MiB)
Everything is Ok

real 124m1.790s
user 136m47.028s
sys 0m23.691s

[mhx]

Well, yes. The main difference here is -md=2536m (and I think that should actually be -md=1536m), setting the dictionary size to 1.5 GiB. However, that is quite pointless in a DwarFS image as it would require a block size of at least 1.5 GiB. DwarFS blocks are usually 16-64 MiB to allow for fast random access (the smaller, the faster).

Because of that, there is absolutely no point of adding "7zip compression" (which isn't even a thing, it's just using LZMA after all) to DwarFS as it would still be limited by the block size. If you increased the block size so your whole source data fits into a single block, you'll be able to get the same compression ration with DwarFS. But again, that's completely pointless.

If you want the best possible compression and do not care about randomly accessing your files as a file system, don't use DwarFS and just use an archiving tool.

If you want to use DwarFS because you care about the "file system" part that's in the name, come to terms with the fact that it most likely won't be able to beat the compression ratio of a well tuned archiver.