mirror of
https://github.com/mhx/dwarfs.git
synced 2025-09-10 04:50:31 -04:00
Add Cromfs comparison
This commit is contained in:
Marcus Holland-Moritz
parent
37244274c4
commit
01107e0d39
167
README.md
167
README.md
@ -18,6 +18,7 @@ A fast high compression read-only file system
|
|||||||
* [With SquashFS](#with-squashfs)
|
* [With SquashFS](#with-squashfs)
|
||||||
* [With SquashFS & xz](#with-squashfs--xz)
|
* [With SquashFS & xz](#with-squashfs--xz)
|
||||||
* [With wimlib](#with-wimlib)
|
* [With wimlib](#with-wimlib)
|
||||||
|
* [With Cromfs](#with-cromfs)
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
@ -865,3 +866,169 @@ However, it's really usable as a file system, even though it's about
|
|||||||
Summary
|
Summary
|
||||||
'dwarfs' ran
|
'dwarfs' ran
|
||||||
4.54 ± 0.73 times faster than 'wimlib'
|
4.54 ± 0.73 times faster than 'wimlib'
|
||||||
|
|
||||||
|
### With Cromfs
|
||||||
|
|
||||||
|
I used [Cromfs](https://bisqwit.iki.fi/source/cromfs.html) in the past
|
||||||
|
for compressed file systems and remember that it did a pretty good job
|
||||||
|
in terms of compression ratio. But it was never fast. However, I didn't
|
||||||
|
quite remember just *how* slow it was until I tried to set up a test.
|
||||||
|
|
||||||
|
Here's a run on the Perl dataset, with the block size set to 16 MiB to
|
||||||
|
match the default of DwarFS, and with additional options suggested to
|
||||||
|
speed up compression:
|
||||||
|
|
||||||
|
$ time mkcromfs -f 16777216 -qq -e -r100000 install perl-install.cromfs
|
||||||
|
Writing perl-install.cromfs...
|
||||||
|
mkcromfs: Automatically enabling --24bitblocknums because it seems possible for this filesystem.
|
||||||
|
Root pseudo file is 108 bytes
|
||||||
|
Inotab spans 0x7f3a18259000..0x7f3a1bfffb9c
|
||||||
|
Root inode spans 0x7f3a205d2948..0x7f3a205d294c
|
||||||
|
Beginning task for Files and directories: Finding identical blocks
|
||||||
|
2163608 reuse opportunities found. 561362 unique blocks. Block table will be 79.4% smaller than without the index search.
|
||||||
|
Beginning task for Files and directories: Blockifying
|
||||||
|
Blockifying: 0.04% (140017/2724970) idx(siz=80423,del=0) rawin(20.97 MB)rawout(20.97 MB)diff(1956 bytes)
|
||||||
|
Termination signalled, cleaning up temporaries
|
||||||
|
|
||||||
|
real 29m9.634s
|
||||||
|
user 201m37.816s
|
||||||
|
sys 2m15.005s
|
||||||
|
|
||||||
|
So it processed 21 MiB out of 48 GiB in half an hour, using almost
|
||||||
|
twice as much CPU resources as DwarFS for the *whole* file system.
|
||||||
|
At this point I decided it's likely not worth waiting (presumably)
|
||||||
|
another month (!) for `mkcromfs` to finish. I double checked that
|
||||||
|
I didn't accidentally build a debugging version, `mkcromfs` was
|
||||||
|
definitely built with `-O3`.
|
||||||
|
|
||||||
|
I then tried once more with a smaller version of the Perl dataset.
|
||||||
|
This only has 20 versions (instead of 1139) of Perl, and obviously
|
||||||
|
a lot less redundancy:
|
||||||
|
|
||||||
|
$ time mkcromfs -f 16777216 -qq -e -r100000 install-small perl-install.cromfs
|
||||||
|
Writing perl-install.cromfs...
|
||||||
|
mkcromfs: Automatically enabling --16bitblocknums because it seems possible for this filesystem.
|
||||||
|
Root pseudo file is 108 bytes
|
||||||
|
Inotab spans 0x7f00e0774000..0x7f00e08410a8
|
||||||
|
Root inode spans 0x7f00b40048f8..0x7f00b40048fc
|
||||||
|
Beginning task for Files and directories: Finding identical blocks
|
||||||
|
25362 reuse opportunities found. 9815 unique blocks. Block table will be 72.1% smaller than without the index search.
|
||||||
|
Beginning task for Files and directories: Blockifying
|
||||||
|
Compressing raw rootdir inode (28 bytes)z=982370,del=2) rawin(641.56 MB)rawout(252.72 MB)diff(388.84 MB)
|
||||||
|
compressed into 35 bytes
|
||||||
|
INOTAB pseudo file is 839.85 kB
|
||||||
|
Inotab inode spans 0x7f00bc036ed8..0x7f00bc036ef4
|
||||||
|
Beginning task for INOTAB: Finding identical blocks
|
||||||
|
0 reuse opportunities found. 13 unique blocks. Block table will be 0.0% smaller than without the index search.
|
||||||
|
Beginning task for INOTAB: Blockifying
|
||||||
|
mkcromfs: Automatically enabling --packedblocks because it is possible for this filesystem.
|
||||||
|
Compressing raw inotab inode (52 bytes)
|
||||||
|
compressed into 58 bytes
|
||||||
|
Compressing 9828 block records (4 bytes each, total 39312 bytes)
|
||||||
|
compressed into 15890 bytes
|
||||||
|
Compressing and writing 16 fblocks...
|
||||||
|
|
||||||
|
16 fblocks were written: 35.31 MB = 13.90 % of 254.01 MB
|
||||||
|
Filesystem size: 35.33 MB = 5.50 % of original 642.22 MB
|
||||||
|
End
|
||||||
|
|
||||||
|
real 27m38.833s
|
||||||
|
user 277m36.208s
|
||||||
|
sys 11m36.945s
|
||||||
|
|
||||||
|
And repeating the same task with `mkdwarfs`:
|
||||||
|
|
||||||
|
$ time mkdwarfs -i install-small -o perl-install-small.dwarfs
|
||||||
|
14:52:09.009618 scanning install-small
|
||||||
|
14:52:09.195087 waiting for background scanners...
|
||||||
|
14:52:09.612164 assigning directory and link inodes...
|
||||||
|
14:52:09.618281 finding duplicate files...
|
||||||
|
14:52:09.718756 saved 267.8 MiB / 611.8 MiB in 22842/26401 duplicate files
|
||||||
|
14:52:09.718837 waiting for inode scanners...
|
||||||
|
14:52:09.926978 assigning device inodes...
|
||||||
|
14:52:09.927745 assigning pipe/socket inodes...
|
||||||
|
14:52:09.928211 building metadata...
|
||||||
|
14:52:09.928293 building blocks...
|
||||||
|
14:52:09.928302 saving names and links...
|
||||||
|
14:52:09.928382 ordering 3559 inodes by similarity...
|
||||||
|
14:52:09.930836 3559 inodes ordered [2.401ms]
|
||||||
|
14:52:09.930891 assigning file inodes...
|
||||||
|
14:52:09.933716 updating name and link indices...
|
||||||
|
14:52:27.051383 waiting for block compression to finish...
|
||||||
|
14:52:27.072944 saving chunks...
|
||||||
|
14:52:27.074108 saving directories...
|
||||||
|
14:52:27.154133 waiting for compression to finish...
|
||||||
|
14:52:40.508238 compressed 611.8 MiB to 25.76 MiB (ratio=0.0420963)
|
||||||
|
14:52:40.525452 filesystem created without errors [31.52s]
|
||||||
|
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
|
||||||
|
waiting for block compression to finish
|
||||||
|
scanned/found: 3334/3334 dirs, 0/0 links, 26401/26401 files
|
||||||
|
original size: 611.8 MiB, dedupe: 267.8 MiB (22842 files), segment: 142.8 MiB
|
||||||
|
filesystem: 201.2 MiB in 13 blocks (9847 chunks, 3559/3559 inodes)
|
||||||
|
compressed filesystem: 13 blocks/25.76 MiB written
|
||||||
|
██████████████████████████████████████████████████████████████████████▏100% |
|
||||||
|
|
||||||
|
real 0m31.553s
|
||||||
|
user 3m21.854s
|
||||||
|
sys 0m3.726s
|
||||||
|
|
||||||
|
So `mkdwarfs` is about 50 times faster than `mkcromfs` and uses 80 times
|
||||||
|
less CPU resources. At the same time, the DwarFS file system is 25% smaller:
|
||||||
|
|
||||||
|
$ ls -l perl-install-small.*fs
|
||||||
|
-rw-r--r-- 1 mhx users 35328512 Dec 8 14:25 perl-install-small.cromfs
|
||||||
|
-rw-r--r-- 1 mhx users 27006735 Dec 8 14:52 perl-install-small.dwarfs
|
||||||
|
|
||||||
|
I noticed that the `blockifying` step that took ages for the full dataset
|
||||||
|
with `mkcromfs` ran substantially faster (in terms of MiB/second) on the
|
||||||
|
smaller dataset, which makes me wonder if there's some quadratic complexity
|
||||||
|
behaviour that's slowing down `mkcromfs`.
|
||||||
|
|
||||||
|
In order to be completely fair, I also ran `mkdwarfs` with `-l 9` to enable
|
||||||
|
LZMA compression (which is what `mkcromfs` uses by default):
|
||||||
|
|
||||||
|
$ time mkdwarfs -i install-small -o perl-install-small-l9.dwarfs -l 9
|
||||||
|
15:05:59.344501 scanning install-small
|
||||||
|
15:05:59.529269 waiting for background scanners...
|
||||||
|
15:05:59.933753 assigning directory and link inodes...
|
||||||
|
15:05:59.938668 finding duplicate files...
|
||||||
|
15:06:00.026974 saved 267.8 MiB / 611.8 MiB in 22842/26401 duplicate files
|
||||||
|
15:06:00.027054 waiting for inode scanners...
|
||||||
|
15:06:00.240184 assigning device inodes...
|
||||||
|
15:06:00.241129 assigning pipe/socket inodes...
|
||||||
|
15:06:00.241723 building metadata...
|
||||||
|
15:06:00.241803 building blocks...
|
||||||
|
15:06:00.241840 saving names and links...
|
||||||
|
15:06:00.241992 ordering 3559 inodes by similarity...
|
||||||
|
15:06:00.246133 3559 inodes ordered [4.057ms]
|
||||||
|
15:06:00.246219 assigning file inodes...
|
||||||
|
15:06:00.248957 updating name and link indices...
|
||||||
|
15:06:19.132473 waiting for block compression to finish...
|
||||||
|
15:06:19.133229 saving chunks...
|
||||||
|
15:06:19.134430 saving directories...
|
||||||
|
15:06:19.192477 waiting for compression to finish...
|
||||||
|
15:06:33.125893 compressed 611.8 MiB to 21.06 MiB (ratio=0.0344202)
|
||||||
|
15:06:33.136930 filesystem created without errors [33.79s]
|
||||||
|
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
|
||||||
|
waiting for block compression to finish
|
||||||
|
scanned/found: 3334/3334 dirs, 0/0 links, 26401/26401 files
|
||||||
|
original size: 611.8 MiB, dedupe: 267.8 MiB (22842 files), segment: 142.8 MiB
|
||||||
|
filesystem: 201.2 MiB in 13 blocks (9847 chunks, 3559/3559 inodes)
|
||||||
|
compressed filesystem: 13 blocks/21.06 MiB written
|
||||||
|
██████████████████████████████████████████████████████████████████████▏100% \
|
||||||
|
|
||||||
|
real 0m33.834s
|
||||||
|
user 3m56.922s
|
||||||
|
sys 0m4.328s
|
||||||
|
|
||||||
|
$ ls -l perl-install-small*.*fs
|
||||||
|
-rw-r--r-- 1 mhx users 22082143 Dec 8 15:06 perl-install-small-l9.dwarfs
|
||||||
|
-rw-r--r-- 1 mhx users 35328512 Dec 8 14:25 perl-install-small.cromfs
|
||||||
|
-rw-r--r-- 1 mhx users 26928161 Dec 8 15:05 perl-install-small.dwarfs
|
||||||
|
|
||||||
|
It only takes 2 seconds longer to build the DwarFS file system with LZMA
|
||||||
|
compression, but reduces the size even further to make it almost 40%
|
||||||
|
smaller than the Cromfs file system.
|
||||||
|
|
||||||
|
I would have added some benchmarks with the Cromfs FUSE driver, but sadly
|
||||||
|
it crashed right upon trying to list the directory after mounting.
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user