TheCloud/dwarfs
1
0
Fork 0
mirror of https://github.com/mhx/dwarfs.git synced 2025-08-04 02:06:22 -04:00
Code Issues Packages Projects Releases Wiki Activity

Update README

Browse Source
This commit is contained in:
Marcus Holland-Moritz 2021-03-04 23:26:27 +01:00
parent 1535ffc85a
commit 635f6840e7
1 changed files with 297 additions and 255 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

552
README.md
View File

@ -163,53 +163,53 @@ A good starting point for apt-based systems is probably:
You can pick either `clang` or `g++`, but at least recent `clang` You can pick either `clang` or `g++`, but at least recent `clang`
versions will produce substantially faster code: versions will produce substantially faster code:
$ hyperfine ./dwarfs_test-* $ hyperfine -L prog $(echo build-*/mkdwarfs | tr ' ' ,) '{prog} --no-progress --log-level warn -i /usr/include -o /dev/null -C null'
Benchmark #1: ./dwarfs_test-clang-O2 Benchmark #1: build-clang-10/mkdwarfs --no-progress --log-level warn -i /usr/include -o /dev/null -C null
Time (mean ± σ): 9.425 s ± 0.049 s [User: 15.724 s, System: 0.773 s] Time (mean ± σ): 6.403 s ± 0.178 s [User: 12.039 s, System: 1.963 s]
Range (min … max): 9.373 s … 9.523 s 10 runs Range (min … max): 6.250 s … 6.701 s 10 runs
Benchmark #2: ./dwarfs_test-clang-O3 Benchmark #2: build-clang-11/mkdwarfs --no-progress --log-level warn -i /usr/include -o /dev/null -C null
Time (mean ± σ): 9.328 s ± 0.045 s [User: 15.593 s, System: 0.791 s] Time (mean ± σ): 6.408 s ± 0.143 s [User: 12.109 s, System: 1.974 s]
Range (min … max): 9.277 s … 9.418 s 10 runs Range (min … max): 6.231 s … 6.617 s 10 runs
Benchmark #3: ./dwarfs_test-gcc-O2 Benchmark #3: build-gcc-10/mkdwarfs --no-progress --log-level warn -i /usr/include -o /dev/null -C null
Time (mean ± σ): 13.798 s ± 0.035 s [User: 20.161 s, System: 0.767 s] Time (mean ± σ): 11.484 s ± 0.245 s [User: 18.487 s, System: 2.071 s]
Range (min … max): 13.731 s … 13.852 s 10 runs Range (min … max): 11.119 s … 11.779 s 10 runs
Benchmark #4: ./dwarfs_test-gcc-O3 Benchmark #4: build-gcc-9/mkdwarfs --no-progress --log-level warn -i /usr/include -o /dev/null -C null
Time (mean ± σ): 13.223 s ± 0.034 s [User: 19.576 s, System: 0.769 s] Time (mean ± σ): 11.443 s ± 0.242 s [User: 18.419 s, System: 2.067 s]
Range (min … max): 13.176 s … 13.278 s 10 runs Range (min … max): 11.177 s … 11.742 s 10 runs
Summary Summary
'./dwarfs_test-clang-O3' ran 'build-clang-10/mkdwarfs --no-progress --log-level warn -i /usr/include -o /dev/null -C null' ran
1.01 ± 0.01 times faster than './dwarfs_test-clang-O2' 1.00 ± 0.04 times faster than 'build-clang-11/mkdwarfs --no-progress --log-level warn -i /usr/include -o /dev/null -C null'
1.42 ± 0.01 times faster than './dwarfs_test-gcc-O3' 1.79 ± 0.06 times faster than 'build-gcc-9/mkdwarfs --no-progress --log-level warn -i /usr/include -o /dev/null -C null'
1.48 ± 0.01 times faster than './dwarfs_test-gcc-O2' 1.79 ± 0.06 times faster than 'build-gcc-10/mkdwarfs --no-progress --log-level warn -i /usr/include -o /dev/null -C null'
$ hyperfine -L prog $(echo mkdwarfs-* | tr ' ' ,) '{prog} --no-progress --log-level warn -i tree -o /dev/null -C null' $ hyperfine build-*/dwarfs_test
Benchmark #1: mkdwarfs-clang-O2 --no-progress --log-level warn -i tree -o /dev/null -C null Benchmark #1: build-clang-10/dwarfs_test
Time (mean ± σ): 4.358 s ± 0.033 s [User: 6.364 s, System: 0.622 s] Time (mean ± σ): 1.789 s ± 0.008 s [User: 2.049 s, System: 0.636 s]
Range (min … max): 4.321 s … 4.408 s 10 runs Range (min … max): 1.775 s … 1.808 s 10 runs
Benchmark #2: mkdwarfs-clang-O3 --no-progress --log-level warn -i tree -o /dev/null -C null Benchmark #2: build-clang-11/dwarfs_test
Time (mean ± σ): 4.282 s ± 0.035 s [User: 6.249 s, System: 0.623 s] Time (mean ± σ): 1.806 s ± 0.011 s [User: 2.053 s, System: 0.660 s]
Range (min … max): 4.244 s … 4.349 s 10 runs Range (min … max): 1.791 s … 1.820 s 10 runs
Benchmark #3: mkdwarfs-gcc-O2 --no-progress --log-level warn -i tree -o /dev/null -C null Benchmark #3: build-gcc-10/dwarfs_test
Time (mean ± σ): 6.212 s ± 0.031 s [User: 8.185 s, System: 0.638 s] Time (mean ± σ): 2.027 s ± 0.004 s [User: 2.270 s, System: 0.797 s]
Range (min … max): 6.159 s … 6.250 s 10 runs Range (min … max): 2.023 s … 2.032 s 10 runs
Benchmark #4: mkdwarfs-gcc-O3 --no-progress --log-level warn -i tree -o /dev/null -C null Benchmark #4: build-gcc-9/dwarfs_test
Time (mean ± σ): 5.740 s ± 0.037 s [User: 7.742 s, System: 0.645 s] Time (mean ± σ): 2.033 s ± 0.005 s [User: 2.278 s, System: 0.796 s]
Range (min … max): 5.685 s … 5.796 s 10 runs Range (min … max): 2.027 s … 2.040 s 10 runs
Summary Summary
'mkdwarfs-clang-O3 --no-progress --log-level warn -i tree -o /dev/null -C null' ran 'build-clang-10/dwarfs_test' ran
1.02 ± 0.01 times faster than 'mkdwarfs-clang-O2 --no-progress --log-level warn -i tree -o /dev/null -C null' 1.01 ± 0.01 times faster than 'build-clang-11/dwarfs_test'
1.34 ± 0.01 times faster than 'mkdwarfs-gcc-O3 --no-progress --log-level warn -i tree -o /dev/null -C null' 1.13 ± 0.01 times faster than 'build-gcc-10/dwarfs_test'
1.45 ± 0.01 times faster than 'mkdwarfs-gcc-O2 --no-progress --log-level warn -i tree -o /dev/null -C null' 1.14 ± 0.01 times faster than 'build-gcc-9/dwarfs_test'
These measurements were made with gcc-9.3.0 and clang-10.0.1. These measurements were made with gcc-9.3.0, gcc-10.2.0, clang-10.0.1 and clang-11.0.1.
### Building ### Building
@ -274,9 +274,10 @@ what can currently be done with the interface.
## Usage ## Usage
Please check out the man pages for [mkdwarfs](doc/mkdwarfs.md) Please check out the man pages for [mkdwarfs](doc/mkdwarfs.md),
and [dwarfs](doc/dwarfs.md). `dwarfsck` will be built and installed [dwarfs](doc/dwarfs.md) and [dwarfsextract](doc/dwarfsextract.md).
as well, but it's still work in progress. `dwarfsck` will be built and installed as well, but it's still
work in progress.
The [dwarfs](doc/dwarfs.md) man page also shows an example for setting The [dwarfs](doc/dwarfs.md) man page also shows an example for setting
up DwarFS with [overlayfs](https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt) up DwarFS with [overlayfs](https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt)
@ -285,11 +286,16 @@ DwarFS image.
## Comparison ## Comparison
### With SquashFS The SquashFS and `xz` tests were all done on an 8 core Intel(R) Xeon(R)
E-2286M CPU @ 2.40GHz with 64 GiB of RAM.
These tests were done on an Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz The wimlib, Cromfs and EROFS tests were done with an older version of
6 core CPU with 64 GiB of RAM. The system was mostly idle during DwarFS on a 6 core Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz with 64 GiB
all of the tests. of RAM.
The systems were mostly idle during all of the tests.
### With SquashFS
The source directory contained **1139 different Perl installations** The source directory contained **1139 different Perl installations**
from 284 distinct releases, a total of 47.65 GiB of data in 1,927,501 from 284 distinct releases, a total of 47.65 GiB of data in 1,927,501
@ -301,9 +307,9 @@ I'm using the same compression type and compression level for
SquashFS that is the default setting for DwarFS: SquashFS that is the default setting for DwarFS:
$ time mksquashfs install perl-install.squashfs -comp zstd -Xcompression-level 22 $ time mksquashfs install perl-install.squashfs -comp zstd -Xcompression-level 22
Parallel mksquashfs: Using 12 processors Parallel mksquashfs: Using 16 processors
Creating 4.0 filesystem on perl-install.squashfs, block size 131072. Creating 4.0 filesystem on perl-install-zstd.squashfs, block size 131072.
[=====================================================================-] 2107401/2107401 100% [=========================================================/] 2107401/2107401 100%
Exportable Squashfs 4.0 filesystem, zstd compressed, data block size 131072 Exportable Squashfs 4.0 filesystem, zstd compressed, data block size 131072
compressed data, compressed metadata, compressed fragments, compressed data, compressed metadata, compressed fragments,
@ -330,84 +336,121 @@ SquashFS that is the default setting for DwarFS:
Number of gids 1 Number of gids 1
users (100) users (100)
real 69m18.427s real 32m54.713s
user 817m15.199s user 501m46.382s
sys 1m38.237s sys 0m58.528s
For DwarFS, I'm sticking to the defaults: For DwarFS, I'm sticking to the defaults:
$ time mkdwarfs -i install -o perl-install.dwarfs $ time mkdwarfs -i install -o perl-install.dwarfs
I 23:54:10.119098 scanning install I 20:22:43.450128 scanning install
I 23:54:32.317096 waiting for background scanners... I 20:22:49.254365 waiting for background scanners...
I 23:55:04.131755 assigning directory and link inodes... I 20:23:01.612414 assigning directory and link inodes...
I 23:55:04.457824 finding duplicate files... I 20:23:01.821212 finding duplicate files...
I 23:55:22.041760 saved 28.2 GiB / 47.65 GiB in 1782826/1927501 duplicate files I 20:23:11.331823 saved 28.2 GiB / 47.65 GiB in 1782826/1927501 duplicate files
I 23:55:22.041862 waiting for inode scanners... I 20:23:11.332706 waiting for inode scanners...
I 23:55:55.365591 assigning device inodes... I 20:23:23.426440 assigning device inodes...
I 23:55:55.423564 assigning pipe/socket inodes... I 20:23:23.462760 assigning pipe/socket inodes...
I 23:55:55.479380 building metadata... I 20:23:23.497944 building metadata...
I 23:55:55.479468 building blocks... I 20:23:23.497991 building blocks...
I 23:55:55.479538 saving names and links... I 20:23:23.498060 saving names and links...
I 23:55:55.479615 ordering 144675 inodes using nilsimsa similarity... I 20:23:23.498127 ordering 144675 inodes using nilsimsa similarity...
I 23:55:55.488995 nilsimsa: depth=20000 (1000), limit=255 I 20:23:23.502987 nilsimsa: depth=20000 (1000), limit=255
I 23:55:55.928990 updating name and link indices... I 20:23:23.688388 updating name and link indices...
I 23:55:56.186375 pre-sorted index (659296 name, 366624 path lookups) [697.3ms] I 20:23:23.851168 pre-sorted index (660176 name, 366179 path lookups) [348.1ms]
I 00:02:11.239104 144675 inodes ordered [375.8s] I 20:25:50.432870 144675 inodes ordered [146.9s]
I 00:02:11.239224 waiting for segmenting/blockifying to finish... I 20:25:50.433477 waiting for segmenting/blockifying to finish...
I 00:06:45.599953 saving chunks... I 20:27:06.432510 segmentation matches: good=459247, bad=6685, total=469664
I 00:06:45.639297 saving directories... I 20:27:06.432582 segmentation collisions: L1=0.007%, L2=0.001% [2233350 hashes]
I 00:06:49.937160 waiting for compression to finish... I 20:27:06.432616 saving chunks...
I 00:08:07.873055 compressed 47.65 GiB to 471.6 MiB (ratio=0.0096655) I 20:27:06.457463 saving directories...
I 00:08:08.512030 filesystem created without errors [838.4s] I 20:27:09.324708 waiting for compression to finish...
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ I 20:28:18.778353 compressed 47.65 GiB to 426.5 MiB (ratio=0.00874147)
I 20:28:19.044470 filesystem created without errors [335.6s]
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
waiting for block compression to finish waiting for block compression to finish
330733 dirs, 0/2440 soft/hard links, 1927501/1927501 files, 0 other 330733 dirs, 0/2440 soft/hard links, 1927501/1927501 files, 0 other
original size: 47.65 GiB, dedupe: 28.2 GiB (1782826 files), segment: 12.6 GiB original size: 47.65 GiB, dedupe: 28.2 GiB (1782826 files), segment: 15.19 GiB
filesystem: 6.846 GiB in 439 blocks (471482 chunks, 144675/144675 inodes) filesystem: 4.261 GiB in 273 blocks (318041 chunks, 144675/144675 inodes)
compressed filesystem: 439 blocks/471.6 MiB written [depth: 20000] compressed filesystem: 273 blocks/426.5 MiB written [depth: 20000]
████████████████████████████████████████████████████████████████████████▏100% \ █████████████████████████████████████████████████████████████████████████▏100% \
real 13m58.556s real 5m35.631s
user 135m40.567s user 80m45.564s
sys 2m59.853s sys 1m54.045s
So in this comparison, `mkdwarfs` is almost 5 times faster than `mksquashfs`. So in this comparison, `mkdwarfs` is almost 6 times faster than `mksquashfs`.
In total CPU time, it actually uses 6 times less CPU resources. In total CPU time, it actually uses more than 6 times less CPU resources.
$ ls -l perl-install.*fs $ ll perl-install.*fs
-rw-r--r-- 1 mhx users 494505722 Dec 30 00:22 perl-install.dwarfs -rw-r--r-- 1 mhx users 447230618 Mar 3 20:28 perl-install.dwarfs
-rw-r--r-- 1 mhx users 4748902400 Nov 25 00:37 perl-install.squashfs -rw-r--r-- 1 mhx users 4748902400 Mar 3 20:10 perl-install.squashfs
In terms of compression ratio, the **DwarFS file system is almost 10 times In terms of compression ratio, the **DwarFS file system is more than 10 times
smaller than the SquashFS file system**. With DwarFS, the content has been smaller than the SquashFS file system**. With DwarFS, the content has been
**compressed down to less than 1% (!) of its original size**. This compression **compressed down to less than 0.9% (!) of its original size**. This compression
ratio only considers the data stored in the individual files, not the actual ratio only considers the data stored in the individual files, not the actual
disk space used. On the original EXT4 file system, according to `du`, the disk space used. On the original XFS file system, according to `du`, the
source folder uses 54 GiB, so **the DwarFS image actually only uses 0.85% of source folder uses 52 GiB, so **the DwarFS image actually only uses 0.8% of
the original space**. the original space**.
When using identical block sizes for both file systems, the difference, Here's another comparison using `lzma` compression instead of `zstd`:
quite expectedly, becomes a lot less dramatic:
$ time mksquashfs install perl-install-1M.squashfs -comp zstd -Xcompression-level 22 -b 1M $ time mksquashfs install perl-install-lzma.squashfs -comp lzma
real 41m55.004s real 13m42.825s
user 340m30.012s user 205m40.851s
sys 1m47.945s sys 3m29.088s
$ time mkdwarfs -i install -o perl-install-1M.dwarfs -S 20 $ time mkdwarfs -i install -o perl-install-lzma.dwarfs -l9
real 24m38.027s real 3m51.163s
user 282m37.305s user 50m30.446s
sys 2m37.558s sys 1m46.203s
$ ls -l perl-install-1M.* $ ll perl-install-lzma.*fs
-rw-r--r-- 1 mhx users 2953052798 Dec 10 18:47 perl-install-1M.dwarfs -rw-r--r-- 1 mhx users 315482627 Mar 3 21:23 perl-install-lzma.dwarfs
-rw-r--r-- 1 mhx users 4198944768 Nov 30 10:05 perl-install-1M.squashfs -rw-r--r-- 1 mhx users 3838406656 Mar 3 20:50 perl-install-lzma.squashfs
It's immediately obvious that the runs are significantly faster and the
resulting images are significantly smaller. Still, `mkdwarfs` is about
**4 times faster** and produces and image that's **12 times smaller** than
the SquashFS image. The DwarFS image is only 0.6% of the original file size.
So why not use `lzma` instead of `zstd` by default? The reason is that `lzma`
is about an order of magnitude slower to decompress than `zstd`. If you're
only accessing data on your compressed filesystem occasionally, this might
not be a big deal, but if you use it extensively, `zstd` will result in
better performance.
The comparisons above are not completely fair. `mksquashfs` by default
uses a block size of 128KiB, whereas `mkdwarfs` uses 16MiB blocks by default,
or even 64MiB blocks with `-l9`. When using identical block sizes for both
file systems, the difference, quite expectedly, becomes a lot less dramatic:
$ time mksquashfs install perl-install-lzma-1M.squashfs -comp lzma -b 1M
real 15m43.319s
user 139m24.533s
sys 0m45.132s
$ time mkdwarfs -i install -o perl-install-lzma-1M.dwarfs -l9 -S20 -B3
real 4m52.694s
user 49m20.611s
sys 6m46.285s
$ ll perl-install*.*fs
-rw-r--r-- 1 mhx users 947407018 Mar 3 22:23 perl-install-lzma-1M.dwarfs
-rw-r--r-- 1 mhx users 3407474688 Mar 3 21:54 perl-install-lzma-1M.squashfs
Even this is *still* not entirely fair, as it uses a feature (`-B3`) that allows
DwarFS to reference file chunks from up to two previous filesystem blocks.
But the point is that this is really where SquashFS tops out, as it doesn't But the point is that this is really where SquashFS tops out, as it doesn't
support larger block sizes. And as you'll see below, the larger blocks that support larger block sizes or back-referencing. And as you'll see below, the
DwarFS is using don't necessarily negatively impact performance. larger blocks that DwarFS is using by default don't necessarily negatively
impact performance.
DwarFS also features an option to recompress an existing file system with DwarFS also features an option to recompress an existing file system with
a different compression algorithm. This can be useful as it allows relatively a different compression algorithm. This can be useful as it allows relatively
@ -415,23 +458,26 @@ fast experimentation with different algorithms and options without requiring
a full rebuild of the file system. For example, recompressing the above file a full rebuild of the file system. For example, recompressing the above file
system with the best possible compression (`-l 9`): system with the best possible compression (`-l 9`):
$ time mkdwarfs --recompress -i perl-install.dwarfs -o perl-lzma.dwarfs -l 9 $ time mkdwarfs --recompress -i perl-install.dwarfs -o perl-lzma-re.dwarfs -l9
20:44:43.738823 filesystem rewritten [385.4s] I 20:28:03.246534 filesystem rewrittenwithout errors [148.3s]
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
filesystem: 6.832 GiB in 438 blocks (0 chunks, 0 inodes) filesystem: 4.261 GiB in 273 blocks (0 chunks, 0 inodes)
compressed filesystem: 438/438 blocks/408.9 MiB written compressed filesystem: 273/273 blocks/372.7 MiB written
█████████████████████████████████████████████████████████████████████▏100% | ████████████████████████████████████████████████████████████████████▏100% \
real 6m25.474s real 2m28.279s
user 73m0.298s user 37m8.825s
sys 1m37.701s sys 0m43.256s
$ ls -l perl-*.dwarfs $ ll perl-*.dwarfs
-rw-r--r-- 1 mhx users 494602224 Dec 10 18:20 perl-install.dwarfs -rw-r--r-- 1 mhx users 447230618 Mar 3 20:28 perl-install.dwarfs
-rw-r--r-- 1 mhx users 428802416 Dec 10 20:44 perl-lzma.dwarfs -rw-r--r-- 1 mhx users 390845518 Mar 4 20:28 perl-lzma-re.dwarfs
-rw-r--r-- 1 mhx users 315482627 Mar 3 21:23 perl-install-lzma.dwarfs
This reduces the file system size by another 13%, pushing the total Note that while the recompressed filesystem is smaller than the original image,
compression ratio to 0.84% (or 0.74% when considering disk usage). it is still a lot bigger than the filesystem we previously build with `-l9`.
The reason is that the recompressed image still uses the same block size, and
the block size cannot be changed by recompressing.
In terms of how fast the file system is when using it, a quick test In terms of how fast the file system is when using it, a quick test
I've done is to freshly mount the filesystem created above and run I've done is to freshly mount the filesystem created above and run
@ -439,100 +485,103 @@ each of the 1139 `perl` executables to print their version.
$ hyperfine -c "umount mnt" -p "umount mnt; dwarfs perl-install.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1" -P procs 5 20 -D 5 "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'" $ hyperfine -c "umount mnt" -p "umount mnt; dwarfs perl-install.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1" -P procs 5 20 -D 5 "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'"
Benchmark #1: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P5 sh -c '$0 -v >/dev/null' Benchmark #1: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P5 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 4.092 s ± 0.031 s [User: 2.183 s, System: 4.355 s] Time (mean ± σ): 1.810 s ± 0.013 s [User: 1.847 s, System: 0.623 s]
Range (min … max): 4.022 s … 4.122 s 10 runs Range (min … max): 1.788 s … 1.825 s 10 runs
Benchmark #2: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null' Benchmark #2: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 2.698 s ± 0.027 s [User: 1.979 s, System: 3.977 s] Time (mean ± σ): 1.333 s ± 0.009 s [User: 1.993 s, System: 0.656 s]
Range (min … max): 2.657 s … 2.732 s 10 runs Range (min … max): 1.321 s … 1.354 s 10 runs
Benchmark #3: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P15 sh -c '$0 -v >/dev/null' Benchmark #3: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P15 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 2.341 s ± 0.029 s [User: 1.883 s, System: 3.794 s] Time (mean ± σ): 1.181 s ± 0.018 s [User: 2.086 s, System: 0.712 s]
Range (min … max): 2.303 s … 2.397 s 10 runs Range (min … max): 1.165 s … 1.214 s 10 runs
Benchmark #4: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null' Benchmark #4: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 2.207 s ± 0.037 s [User: 1.818 s, System: 3.673 s] Time (mean ± σ): 1.149 s ± 0.015 s [User: 2.128 s, System: 0.781 s]
Range (min … max): 2.163 s … 2.278 s 10 runs Range (min … max): 1.136 s … 1.186 s 10 runs
These timings are for *initial* runs on a freshly mounted file system, These timings are for *initial* runs on a freshly mounted file system,
running 5, 10, 15 and 20 processes in parallel. 2.2 seconds means that running 5, 10, 15 and 20 processes in parallel. 1.1 seconds means that
it takes only about 2 milliseconds per Perl binary. it takes only about 1 millisecond per Perl binary.
Following are timings for *subsequent* runs, both on DwarFS (at `mnt`) Following are timings for *subsequent* runs, both on DwarFS (at `mnt`)
and the original EXT4 (at `install`). DwarFS is around 15% slower here: and the original XFS (at `install`). DwarFS is around 15% slower here:
$ hyperfine -P procs 10 20 -D 10 -w1 "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'" "ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'" $ hyperfine -P procs 10 20 -D 10 -w1 "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'" "ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'"
Benchmark #1: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null' Benchmark #1: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 655.8 ms ± 5.5 ms [User: 1.716 s, System: 2.784 s] Time (mean ± σ): 347.0 ms ± 7.2 ms [User: 1.755 s, System: 0.452 s]
Range (min … max): 647.6 ms … 664.3 ms 10 runs Range (min … max): 341.3 ms … 365.2 ms 10 runs
Benchmark #2: ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null' Benchmark #2: ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 583.9 ms ± 5.0 ms [User: 1.715 s, System: 2.773 s] Time (mean ± σ): 302.5 ms ± 3.3 ms [User: 1.656 s, System: 0.377 s]
Range (min … max): 577.0 ms … 592.0 ms 10 runs Range (min … max): 297.1 ms … 308.7 ms 10 runs
Benchmark #3: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null' Benchmark #3: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 638.2 ms ± 10.7 ms [User: 1.667 s, System: 2.736 s] Time (mean ± σ): 342.2 ms ± 4.1 ms [User: 1.766 s, System: 0.451 s]
Range (min … max): 629.1 ms … 658.4 ms 10 runs Range (min … max): 336.0 ms … 349.7 ms 10 runs
Benchmark #4: ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null' Benchmark #4: ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 567.0 ms ± 3.2 ms [User: 1.684 s, System: 2.719 s] Time (mean ± σ): 302.0 ms ± 3.0 ms [User: 1.659 s, System: 0.374 s]
Range (min … max): 561.5 ms … 570.5 ms 10 runs Range (min … max): 297.0 ms … 305.4 ms 10 runs
Summary
'ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'' ran
1.00 ± 0.01 times faster than 'ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null''
1.13 ± 0.02 times faster than 'ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null''
1.15 ± 0.03 times faster than 'ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null''
Using the lzma-compressed file system, the metrics for *initial* runs look Using the lzma-compressed file system, the metrics for *initial* runs look
considerably worse: considerably worse (about an order of magnitude):
$ hyperfine -c "umount mnt" -p "umount mnt; dwarfs perl-lzma.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1" -P procs 5 20 -D 5 "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'" $ hyperfine -c "umount mnt" -p "umount mnt; dwarfs perl-install-lzma.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1" -P procs 5 20 -D 5 "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'"
Benchmark #1: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P5 sh -c '$0 -v >/dev/null' Benchmark #1: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P5 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 20.170 s ± 0.138 s [User: 2.379 s, System: 4.713 s] Time (mean ± σ): 10.660 s ± 0.057 s [User: 1.952 s, System: 0.729 s]
Range (min … max): 19.968 s … 20.390 s 10 runs Range (min … max): 10.615 s … 10.811 s 10 runs
Benchmark #2: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null' Benchmark #2: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 12.960 s ± 0.105 s [User: 2.130 s, System: 4.272 s] Time (mean ± σ): 9.092 s ± 0.021 s [User: 1.979 s, System: 0.680 s]
Range (min … max): 12.807 s … 13.122 s 10 runs Range (min … max): 9.059 s … 9.126 s 10 runs
Benchmark #3: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P15 sh -c '$0 -v >/dev/null' Benchmark #3: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P15 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 11.495 s ± 0.141 s [User: 1.995 s, System: 4.100 s] Time (mean ± σ): 9.012 s ± 0.188 s [User: 2.077 s, System: 0.702 s]
Range (min … max): 11.324 s … 11.782 s 10 runs Range (min … max): 8.839 s … 9.277 s 10 runs
Benchmark #4: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null' Benchmark #4: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
Time (mean ± σ): 11.238 s ± 0.100 s [User: 1.894 s, System: 3.855 s] Time (mean ± σ): 9.004 s ± 0.298 s [User: 2.134 s, System: 0.736 s]
Range (min … max): 11.119 s … 11.371 s 10 runs Range (min … max): 8.611 s … 9.555 s 10 runs
So you might want to consider using zstd instead of lzma if you'd So you might want to consider using `zstd` instead of `lzma` if you'd
like to optimize for file system performance. It's also the default like to optimize for file system performance. It's also the default
compression used by `mkdwarfs`. compression used by `mkdwarfs`.
On a different system, Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz, Now here's a comparison with the SquashFS filesystem:
with 4 cores, I did more tests with both SquashFS and DwarFS
(just because on the 6 core box my kernel didn't have support
for zstd in SquashFS):
hyperfine -c 'sudo umount /tmp/perl/install' -p 'umount /tmp/perl/install; dwarfs perl-install.dwarfs /tmp/perl/install -o cachesize=1g -o workers=4; sleep 1' -n dwarfs-zstd "ls -1 /tmp/perl/install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '\$0 -v >/dev/null'" -p 'sudo umount /tmp/perl/install; sudo mount -t squashfs perl-install.squashfs /tmp/perl/install; sleep 1' -n squashfs-zstd "ls -1 /tmp/perl/install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '\$0 -v >/dev/null'" $ hyperfine -c 'sudo umount mnt' -p 'umount mnt; dwarfs perl-install.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1' -n dwarfs-zstd "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '\$0 -v >/dev/null'" -p 'sudo umount mnt; sudo mount -t squashfs perl-install.squashfs mnt; sleep 1' -n squashfs-zstd "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '\$0 -v >/dev/null'"
Benchmark #1: dwarfs-zstd Benchmark #1: dwarfs-zstd
Time (mean ± σ): 2.071 s ± 0.372 s [User: 1.727 s, System: 2.866 s] Time (mean ± σ): 1.151 s ± 0.015 s [User: 2.147 s, System: 0.769 s]
Range (min … max): 1.711 s … 2.532 s 10 runs Range (min … max): 1.118 s … 1.174 s 10 runs
Benchmark #2: squashfs-zstd Benchmark #2: squashfs-zstd
Time (mean ± σ): 3.668 s ± 0.070 s [User: 2.173 s, System: 21.287 s] Time (mean ± σ): 6.733 s ± 0.007 s [User: 3.188 s, System: 17.015 s]
Range (min … max): 3.616 s … 3.846 s 10 runs Range (min … max): 6.721 s … 6.743 s 10 runs
Summary Summary
'dwarfs-zstd' ran 'dwarfs-zstd' ran
1.77 ± 0.32 times faster than 'squashfs-zstd' 5.85 ± 0.08 times faster than 'squashfs-zstd'
So DwarFS is almost twice as fast as SquashFS. But what's more, So DwarFS is almost six times faster than SquashFS. But what's more,
SquashFS also uses significantly more CPU power. However, the numbers SquashFS also uses significantly more CPU power. However, the numbers
shown above for DwarFS obviously don't include the time spent in the shown above for DwarFS obviously don't include the time spent in the
`dwarfs` process, so I repeated the test outside of hyperfine: `dwarfs` process, so I repeated the test outside of hyperfine:
$ time dwarfs perl-install.dwarfs /tmp/perl/install -o cachesize=1g -o workers=4 -f $ time dwarfs perl-install.dwarfs mnt -o cachesize=1g -o workers=4 -f
real 0m8.463s real 0m4.569s
user 0m3.821s user 0m2.154s
sys 0m2.117s sys 0m1.846s
So in total, DwarFS was using 10.5 seconds of CPU time, whereas So in total, DwarFS was using 5.7 seconds of CPU time, whereas
SquashFS was using 23.5 seconds, more than twice as much. Ignore SquashFS was using 20.2 seconds, almost four times as much. Ignore
the 'real' time, this is only how long it took me to unmount the the 'real' time, this is only how long it took me to unmount the
file system again after mounting it. file system again after mounting it.
@ -557,7 +606,7 @@ I wrote a little script to be able to run multiple builds in parallel:
set -eu set -eu
perl=$1 perl=$1
dir=$(echo "$perl" | cut -d/ --output-delimiter=- -f5,6) dir=$(echo "$perl" | cut -d/ --output-delimiter=- -f5,6)
rsync -a Tie-Hash-Indexed-0.08/ $dir/ rsync -a Tie-Hash-Indexed/ $dir/
cd $dir cd $dir
$1 Makefile.PL >/dev/null 2>&1 $1 Makefile.PL >/dev/null 2>&1
make test >/dev/null 2>&1 make test >/dev/null 2>&1
@ -566,34 +615,35 @@ rm -rf $dir
echo $perl echo $perl
``` ```
The following command will run up to 8 builds in parallel on the 4 core The following command will run up to 16 builds in parallel on the 8 core
i7 CPU, including debug, optimized and threaded versions of all Perl Xeon CPU, including debug, optimized and threaded versions of all Perl
releases between 5.10.0 and 5.33.3, a total of 624 `perl` installations: releases between 5.10.0 and 5.33.3, a total of 624 `perl` installations:
$ time ls -1 /tmp/perl/install/*/perl-5.??.?/bin/perl5* | sort -t / -k 8 | xargs -d $'\n' -P 8 -n 1 ./build.sh $ time ls -1 /tmp/perl/install/*/perl-5.??.?/bin/perl5* | sort -t / -k 8 | xargs -d $'\n' -P 16 -n 1 ./build.sh
Tests were done with a cleanly mounted file system to make sure the caches Tests were done with a cleanly mounted file system to make sure the caches
were empty. `ccache` was primed to make sure all compiler runs could be were empty. `ccache` was primed to make sure all compiler runs could be
satisfied from the cache. With SquashFS, the timing was: satisfied from the cache. With SquashFS, the timing was:
real 3m17.182s real 0m52.385s
user 20m54.064s user 8m10.333s
sys 4m16.907s sys 4m10.056s
And with DwarFS: And with DwarFS:
real 3m14.402s real 0m50.469s
user 19m42.984s user 9m22.597s
sys 2m49.292s sys 1m18.469s
So, frankly, not much of a difference. The `dwarfs` process itself used: So, frankly, not much of a difference, with DwarFS being just a bit faster.
The `dwarfs` process itself used:
real 4m23.151s real 0m56.686s
user 0m25.036s user 0m18.857s
sys 0m35.216s sys 0m21.058s
So again, DwarFS used less raw CPU power, but in terms of wallclock time, So again, DwarFS used less raw CPU power overall, but in terms of wallclock
the difference is really marginal. time, the difference is really marginal.
### With SquashFS & xz ### With SquashFS & xz
@ -602,60 +652,60 @@ a recent Raspberry Pi OS release. This file system also contains device inodes,
so in order to preserve those, we pass `--with-devices` to `mkdwarfs`: so in order to preserve those, we pass `--with-devices` to `mkdwarfs`:
$ time sudo mkdwarfs -i raspbian -o raspbian.dwarfs --with-devices $ time sudo mkdwarfs -i raspbian -o raspbian.dwarfs --with-devices
20:49:45.099221 scanning raspbian I 21:30:29.812562 scanning raspbian
20:49:45.395243 waiting for background scanners... I 21:30:29.908984 waiting for background scanners...
20:49:46.019979 assigning directory and link inodes... I 21:30:30.217446 assigning directory and link inodes...
20:49:46.035099 finding duplicate files... I 21:30:30.221941 finding duplicate files...
20:49:46.148490 saved 31.05 MiB / 1007 MiB in 1617/34582 duplicate files I 21:30:30.288099 saved 31.05 MiB / 1007 MiB in 1617/34582 duplicate files
20:49:46.149221 waiting for inode scanners... I 21:30:30.288143 waiting for inode scanners...
20:49:48.518179 assigning device inodes... I 21:30:31.393710 assigning device inodes...
20:49:48.519512 assigning pipe/socket inodes... I 21:30:31.394481 assigning pipe/socket inodes...
20:49:48.520322 building metadata... I 21:30:31.395196 building metadata...
20:49:48.520425 building blocks... I 21:30:31.395230 building blocks...
20:49:48.520473 saving names and links... I 21:30:31.395291 saving names and links...
20:49:48.520568 ordering 32965 inodes using nilsimsa similarity... I 21:30:31.395374 ordering 32965 inodes using nilsimsa similarity...
20:49:48.522323 nilsimsa: depth=20000, limit=255 I 21:30:31.396254 nilsimsa: depth=20000 (1000), limit=255
20:49:48.554803 updating name and link indices... I 21:30:31.407967 pre-sorted index (46431 name, 2206 path lookups) [11.66ms]
20:49:48.577389 pre-sorted index (55243 name, 26489 path lookups) [54.95ms] I 21:30:31.410089 updating name and link indices...
20:50:55.921085 32965 inodes ordered [67.4s] I 21:30:38.178505 32965 inodes ordered [6.783s]
20:50:55.921179 waiting for segmenting/blockifying to finish... I 21:30:38.179417 waiting for segmenting/blockifying to finish...
20:51:02.372233 saving chunks... I 21:31:06.248304 saving chunks...
20:51:02.376389 saving directories... I 21:31:06.251998 saving directories...
20:51:02.492263 waiting for compression to finish... I 21:31:06.402559 waiting for compression to finish...
20:51:31.098179 compressed 1007 MiB to 286.6 MiB (ratio=0.284714) I 21:31:16.425563 compressed 1007 MiB to 287 MiB (ratio=0.285036)
20:51:31.140186 filesystem created without errors [106s] I 21:31:16.464772 filesystem created without errors [46.65s]
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
waiting for block compression to finish waiting for block compression to finish
4435 dirs, 5908/473 soft/hard links, 34582/34582 files, 7 other 4435 dirs, 5908/0 soft/hard links, 34582/34582 files, 7 other
original size: 1007 MiB, dedupe: 31.05 MiB (1617 files), segment: 46.9 MiB original size: 1007 MiB, dedupe: 31.05 MiB (1617 files), segment: 47.23 MiB
filesystem: 928.7 MiB in 59 blocks (39117 chunks, 32965/32965 inodes) filesystem: 928.4 MiB in 59 blocks (38890 chunks, 32965/32965 inodes)
compressed filesystem: 59 blocks/286.6 MiB written compressed filesystem: 59 blocks/287 MiB written [depth: 20000]
███████████████████████████████████████████████████████████████████▏100% | ████████████████████████████████████████████████████████████████████▏100% |
real 1m46.153s real 0m46.711s
user 18m7.973s user 10m39.038s
sys 0m16.013s sys 0m8.123s
Again, SquashFS uses the same compression options: Again, SquashFS uses the same compression options:
$ time sudo mksquashfs raspbian raspbian.squashfs -comp zstd -Xcompression-level 22 $ time sudo mksquashfs raspbian raspbian.squashfs -comp zstd -Xcompression-level 22
Parallel mksquashfs: Using 12 processors Parallel mksquashfs: Using 16 processors
Creating 4.0 filesystem on raspbian.squashfs, block size 131072. Creating 4.0 filesystem on raspbian.squashfs, block size 131072.
[====================================================================-] 38644/38644 100% [===============================================================\] 39232/39232 100%
Exportable Squashfs 4.0 filesystem, zstd compressed, data block size 131072 Exportable Squashfs 4.0 filesystem, zstd compressed, data block size 131072
compressed data, compressed metadata, compressed fragments, compressed data, compressed metadata, compressed fragments,
compressed xattrs, compressed ids compressed xattrs, compressed ids
duplicates are removed duplicates are removed
Filesystem size 371931.65 Kbytes (363.21 Mbytes) Filesystem size 371934.50 Kbytes (363.22 Mbytes)
36.89% of uncompressed filesystem size (1008353.15 Kbytes) 35.98% of uncompressed filesystem size (1033650.60 Kbytes)
Inode table size 398565 bytes (389.22 Kbytes) Inode table size 399913 bytes (390.54 Kbytes)
26.61% of uncompressed inode table size (1497593 bytes) 26.53% of uncompressed inode table size (1507581 bytes)
Directory table size 408794 bytes (399.21 Kbytes) Directory table size 408749 bytes (399.17 Kbytes)
42.28% of uncompressed directory table size (966980 bytes) 42.31% of uncompressed directory table size (966174 bytes)
Number of duplicate files found 1145 Number of duplicate files found 1618
Number of inodes 44459 Number of inodes 44932
Number of files 34109 Number of files 34582
Number of fragments 3290 Number of fragments 3290
Number of symbolic links 5908 Number of symbolic links 5908
Number of device nodes 7 Number of device nodes 7
@ -666,9 +716,9 @@ Again, SquashFS uses the same compression options:
Number of uids 5 Number of uids 5
root (0) root (0)
mhx (1000) mhx (1000)
logitechmediaserver (103) unknown (103)
shutdown (6) shutdown (6)
x2goprint (106) unknown (106)
Number of gids 15 Number of gids 15
root (0) root (0)
unknown (109) unknown (109)
@ -686,61 +736,53 @@ Again, SquashFS uses the same compression options:
adm (4) adm (4)
mem (8) mem (8)
real 1m54.997s real 0m50.124s
user 18m32.386s user 9m41.708s
sys 0m2.627s sys 0m1.727s
The difference in speed is almost negligible. SquashFS is just a bit The difference in speed is almost negligible. SquashFS is just a bit
slower here. In terms of compression, the difference also isn't huge: slower here. In terms of compression, the difference also isn't huge:
$ ls -lh raspbian.* *.xz $ ls -lh raspbian.* *.xz
-rw-r--r-- 1 root root 287M Dec 10 20:51 raspbian.dwarfs -rw-r--r-- 1 mhx users 297M Mar 4 21:32 2020-08-20-raspios-buster-armhf-lite.img.xz
-rw-r--r-- 1 root root 364M Dec 9 22:31 raspbian.squashfs -rw-r--r-- 1 root root 287M Mar 4 21:31 raspbian.dwarfs
-rw-r--r-- 1 mhx users 297M Aug 20 12:47 2020-08-20-raspios-buster-armhf-lite.img.xz -rw-r--r-- 1 root root 364M Mar 4 21:33 raspbian.squashfs
Interestingly, `xz` actually can't compress the whole original image Interestingly, `xz` actually can't compress the whole original image
better than DwarFS. better than DwarFS.
We can even again try to increase the DwarFS compression level: We can even again try to increase the DwarFS compression level:
$ time mkdwarfs -i raspbian.dwarfs -o raspbian-9.dwarfs -l 9 --recompress $ time sudo mkdwarfs -i raspbian -o raspbian-9.dwarfs --with-devices -l9
20:55:34.416488 filesystem rewritten [69.79s]
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
filesystem: 928.7 MiB in 59 blocks (0 chunks, 0 inodes)
compressed filesystem: 59/59 blocks/257.7 MiB written
██████████████████████████████████████████████████████████████████▏100% \
real 1m9.879s real 0m54.161s
user 12m52.376s user 8m40.109s
sys 0m14.315s sys 0m7.101s
Now that actually gets the DwarFS image size well below that of the Now that actually gets the DwarFS image size well below that of the
`xz` archive: `xz` archive:
$ ls -lh raspbian-9.dwarfs *.xz $ ls -lh raspbian-9.dwarfs *.xz
-rw-r--r-- 1 mhx users 258M Dec 10 20:55 raspbian-9.dwarfs -rw-r--r-- 1 root root 244M Mar 4 21:36 raspbian-9.dwarfs
-rw-r--r-- 1 mhx users 297M Aug 20 12:47 2020-08-20-raspios-buster-armhf-lite.img.xz -rw-r--r-- 1 mhx users 297M Mar 4 21:32 2020-08-20-raspios-buster-armhf-lite.img.xz
However, if you actually build a tarball and compress that (instead of Even if you actually build a tarball and compress that (instead of
compressing the EXT4 file system), `xz` is, unsurprisingly, able to take compressing the EXT4 file system itself), `xz` isn't quite able to
the lead again: match the DwarFS image size:
$ time sudo tar cf - raspbian | xz -9e -vT 0 >raspbian.tar.xz $ time sudo tar cf - raspbian | xz -9 -vT 0 >raspbian.tar.xz
100 % 245.9 MiB / 1,012.3 MiB = 0.243 5.4 MiB/s 3:07 100 % 246.9 MiB / 1,037.2 MiB = 0.238 13 MiB/s 1:18
real 3m8.088s real 1m18.226s
user 14m16.519s user 6m35.381s
sys 0m5.843s sys 0m2.205s
$ ls -lh raspbian.tar.xz $ ls -lh raspbian.tar.xz
-rw-r--r-- 1 mhx users 246M Nov 30 00:16 raspbian.tar.xz -rw-r--r-- 1 mhx users 247M Mar 4 21:40 raspbian.tar.xz
In summary, DwarFS can get pretty close to an `xz` compressed tarball In summary, DwarFS can even outperform an `xz` compressed tarball in
in terms of size. It's also almsot three times faster to build the file terms of size. It's also significantly faster to build the file
system than to build the tarball. At the same time, SquashFS really system than to build the tarball.
isn't that much worse. It's really the cases where you *know* upfront
that your data is highly redundant where DwarFS can play out its full
strength.
### With wimlib ### With wimlib