diff --git a/README.md b/README.md
index a1a96b2b..16146dc0 100644
--- a/README.md
+++ b/README.md
@@ -6,22 +6,22 @@ A fast high compression read-only file system
## Table of contents
-* [Overview](#overview)
-* [History](#history)
-* [Building and Installing](#building-and-installing)
- * [Dependencies](#dependencies)
- * [Building](#building)
- * [Installing](#installing)
- * [Experimental Python Scripting Support](#experimental-python-scripting-support)
-* [Usage](#usage)
-* [Comparison](#comparison)
- * [With SquashFS](#with-squashfs)
- * [With SquashFS & xz](#with-squashfs--xz)
- * [With lrzip](#with-lrzip)
- * [With zpaq](#with-zpaq)
- * [With wimlib](#with-wimlib)
- * [With Cromfs](#with-cromfs)
- * [With EROFS](#with-erofs)
+- [Overview](#overview)
+- [History](#history)
+- [Building and Installing](#building-and-installing)
+ - [Dependencies](#dependencies)
+ - [Building](#building)
+ - [Installing](#installing)
+ - [Experimental Python Scripting Support](#experimental-python-scripting-support)
+- [Usage](#usage)
+- [Comparison](#comparison)
+ - [With SquashFS](#with-squashfs)
+ - [With SquashFS & xz](#with-squashfs--xz)
+ - [With lrzip](#with-lrzip)
+ - [With zpaq](#with-zpaq)
+ - [With wimlib](#with-wimlib)
+ - [With Cromfs](#with-cromfs)
+ - [With EROFS](#with-erofs)
## Overview
@@ -45,20 +45,20 @@ less CPU resources.
Distinct features of DwarFS are:
-* Clustering of files by similarity using a similarity hash function.
+- Clustering of files by similarity using a similarity hash function.
This makes it easier to exploit the redundancy across file boundaries.
-* Segmentation analysis across file system blocks in order to reduce
+- Segmentation analysis across file system blocks in order to reduce
the size of the uncompressed file system. This saves memory when
using the compressed file system and thus potentially allows for
higher cache hit rates as more data can be kept in the cache.
-* Highly multi-threaded implementation. Both the file
+- Highly multi-threaded implementation. Both the file
[system creation tool](doc/mkdwarfs.md) as well as the
[FUSE driver](doc/dwarfs.md) are able to make good use of the
many cores of your system.
-* Optional experimental Python scripting support to provide custom
+- Optional experimental Python scripting support to provide custom
filtering and ordering functionality.
## History
@@ -129,38 +129,40 @@ will be automatically resolved if you build with tests.
A good starting point for apt-based systems is probably:
- $ apt install \
- g++ \
- clang \
- cmake \
- make \
- bison \
- flex \
- ronn \
- fuse3 \
- pkg-config \
- binutils-dev \
- libarchive-dev \
- libboost-context-dev \
- libboost-filesystem-dev \
- libboost-program-options-dev \
- libboost-python-dev \
- libboost-regex-dev \
- libboost-system-dev \
- libboost-thread-dev \
- libevent-dev \
- libjemalloc-dev \
- libdouble-conversion-dev \
- libiberty-dev \
- liblz4-dev \
- liblzma-dev \
- libssl-dev \
- libunwind-dev \
- libdwarf-dev \
- libelf-dev \
- libfmt-dev \
- libfuse3-dev \
- libgoogle-glog-dev
+```
+$ apt install \
+ g++ \
+ clang \
+ cmake \
+ make \
+ bison \
+ flex \
+ ronn \
+ fuse3 \
+ pkg-config \
+ binutils-dev \
+ libarchive-dev \
+ libboost-context-dev \
+ libboost-filesystem-dev \
+ libboost-program-options-dev \
+ libboost-python-dev \
+ libboost-regex-dev \
+ libboost-system-dev \
+ libboost-thread-dev \
+ libevent-dev \
+ libjemalloc-dev \
+ libdouble-conversion-dev \
+ libiberty-dev \
+ liblz4-dev \
+ liblzma-dev \
+ libssl-dev \
+ libunwind-dev \
+ libdwarf-dev \
+ libelf-dev \
+ libfmt-dev \
+ libfuse3-dev \
+ libgoogle-glog-dev
+```
Note that when building with `gcc`, the optimization level will be
set to `-O2` instead of the CMake default of `-O3` for release
@@ -168,30 +170,37 @@ builds. At least with versions up to `gcc-10`, the `-O3` build is
[up to 70% slower](https://github.com/mhx/dwarfs/issues/14) than a
build with `-O2`.
-
### Building
Firstly, either clone the repository...
- $ git clone --recurse-submodules https://github.com/mhx/dwarfs
- $ cd dwarfs
+```
+$ git clone --recurse-submodules https://github.com/mhx/dwarfs
+$ cd dwarfs
+```
...or unpack the release archive:
- $ tar xvf dwarfs-x.y.z.tar.bz2
- $ cd dwarfs-x.y.z
+```
+$ tar xvf dwarfs-x.y.z.tar.bz2
+$ cd dwarfs-x.y.z
+```
Once all dependencies have been installed, you can build DwarFS
using:
- $ mkdir build
- $ cd build
- $ cmake .. -DWITH_TESTS=1
- $ make -j$(nproc)
+```
+$ mkdir build
+$ cd build
+$ cmake .. -DWITH_TESTS=1
+$ make -j$(nproc)
+```
You can then run tests with:
- $ make test
+```
+$ make test
+```
All binaries use [jemalloc](https://github.com/jemalloc/jemalloc)
as a memory allocator by default, as it is typically uses much less
@@ -203,7 +212,9 @@ To disable the use of `jemalloc`, pass `-DUSE_JEMALLOC=0` on the
Installing is as easy as:
- $ sudo make install
+```
+$ sudo make install
+```
Though you don't have to install the tools to play with them.
@@ -212,13 +223,17 @@ Though you don't have to install the tools to play with them.
You can build `mkdwarfs` with experimental support for Python
scripting:
- $ cmake .. -DWITH_TESTS=1 -DWITH_PYTHON=1
+```
+$ cmake .. -DWITH_TESTS=1 -DWITH_PYTHON=1
+```
This also requires Boost.Python. If you have multiple Python
versions installed, you can explicitly specify the version to
build against:
- $ cmake .. -DWITH_TESTS=1 -DWITH_PYTHON=1 -DWITH_PYTHON_VERSION=3.8
+```
+$ cmake .. -DWITH_TESTS=1 -DWITH_PYTHON=1 -DWITH_PYTHON_VERSION=3.8
+```
Note that only Python 3 is supported. You can take a look at
[scripts/example.py](scripts/example.py) to get an idea for
@@ -259,87 +274,93 @@ NVME drive, so most of its contents were likely cached.
I'm using the same compression type and compression level for
SquashFS that is the default setting for DwarFS:
- $ time mksquashfs install perl-install.squashfs -comp zstd -Xcompression-level 22
- Parallel mksquashfs: Using 16 processors
- Creating 4.0 filesystem on perl-install-zstd.squashfs, block size 131072.
- [=========================================================/] 2107401/2107401 100%
-
- Exportable Squashfs 4.0 filesystem, zstd compressed, data block size 131072
- compressed data, compressed metadata, compressed fragments,
- compressed xattrs, compressed ids
- duplicates are removed
- Filesystem size 4637597.63 Kbytes (4528.90 Mbytes)
- 9.29% of uncompressed filesystem size (49922299.04 Kbytes)
- Inode table size 19100802 bytes (18653.13 Kbytes)
- 26.06% of uncompressed inode table size (73307702 bytes)
- Directory table size 19128340 bytes (18680.02 Kbytes)
- 46.28% of uncompressed directory table size (41335540 bytes)
- Number of duplicate files found 1780387
- Number of inodes 2255794
- Number of files 1925061
- Number of fragments 28713
- Number of symbolic links 0
- Number of device nodes 0
- Number of fifo nodes 0
- Number of socket nodes 0
- Number of directories 330733
- Number of ids (unique uids + gids) 2
- Number of uids 1
- mhx (1000)
- Number of gids 1
- users (100)
-
- real 32m54.713s
- user 501m46.382s
- sys 0m58.528s
+```
+$ time mksquashfs install perl-install.squashfs -comp zstd -Xcompression-level 22
+Parallel mksquashfs: Using 16 processors
+Creating 4.0 filesystem on perl-install-zstd.squashfs, block size 131072.
+[=========================================================/] 2107401/2107401 100%
+
+Exportable Squashfs 4.0 filesystem, zstd compressed, data block size 131072
+ compressed data, compressed metadata, compressed fragments,
+ compressed xattrs, compressed ids
+ duplicates are removed
+Filesystem size 4637597.63 Kbytes (4528.90 Mbytes)
+ 9.29% of uncompressed filesystem size (49922299.04 Kbytes)
+Inode table size 19100802 bytes (18653.13 Kbytes)
+ 26.06% of uncompressed inode table size (73307702 bytes)
+Directory table size 19128340 bytes (18680.02 Kbytes)
+ 46.28% of uncompressed directory table size (41335540 bytes)
+Number of duplicate files found 1780387
+Number of inodes 2255794
+Number of files 1925061
+Number of fragments 28713
+Number of symbolic links 0
+Number of device nodes 0
+Number of fifo nodes 0
+Number of socket nodes 0
+Number of directories 330733
+Number of ids (unique uids + gids) 2
+Number of uids 1
+ mhx (1000)
+Number of gids 1
+ users (100)
+
+real 32m54.713s
+user 501m46.382s
+sys 0m58.528s
+```
For DwarFS, I'm sticking to the defaults:
- $ time mkdwarfs -i install -o perl-install.dwarfs
- I 11:33:33.310931 scanning install
- I 11:33:39.026712 waiting for background scanners...
- I 11:33:50.681305 assigning directory and link inodes...
- I 11:33:50.888441 finding duplicate files...
- I 11:34:01.120800 saved 28.2 GiB / 47.65 GiB in 1782826/1927501 duplicate files
- I 11:34:01.122608 waiting for inode scanners...
- I 11:34:12.839065 assigning device inodes...
- I 11:34:12.875520 assigning pipe/socket inodes...
- I 11:34:12.910431 building metadata...
- I 11:34:12.910524 building blocks...
- I 11:34:12.910594 saving names and links...
- I 11:34:12.910691 bloom filter size: 32 KiB
- I 11:34:12.910760 ordering 144675 inodes using nilsimsa similarity...
- I 11:34:12.915555 nilsimsa: depth=20000 (1000), limit=255
- I 11:34:13.052525 updating name and link indices...
- I 11:34:13.276233 pre-sorted index (660176 name, 366179 path lookups) [360.6ms]
- I 11:35:44.039375 144675 inodes ordered [91.13s]
- I 11:35:44.041427 waiting for segmenting/blockifying to finish...
- I 11:37:38.823902 bloom filter reject rate: 96.017% (TPR=0.244%, lookups=4740563665)
- I 11:37:38.823963 segmentation matches: good=454708, bad=6819, total=464247
- I 11:37:38.824005 segmentation collisions: L1=0.008%, L2=0.000% [2233254 hashes]
- I 11:37:38.824038 saving chunks...
- I 11:37:38.860939 saving directories...
- I 11:37:41.318747 waiting for compression to finish...
- I 11:38:56.046809 compressed 47.65 GiB to 430.9 MiB (ratio=0.00883101)
- I 11:38:56.304922 filesystem created without errors [323s]
- ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
- waiting for block compression to finish
- 330733 dirs, 0/2440 soft/hard links, 1927501/1927501 files, 0 other
- original size: 47.65 GiB, dedupe: 28.2 GiB (1782826 files), segment: 15.19 GiB
- filesystem: 4.261 GiB in 273 blocks (319178 chunks, 144675/144675 inodes)
- compressed filesystem: 273 blocks/430.9 MiB written [depth: 20000]
- █████████████████████████████████████████████████████████████████████████████▏100% |
-
- real 5m23.030s
- user 78m7.554s
- sys 1m47.968s
+```
+$ time mkdwarfs -i install -o perl-install.dwarfs
+I 11:33:33.310931 scanning install
+I 11:33:39.026712 waiting for background scanners...
+I 11:33:50.681305 assigning directory and link inodes...
+I 11:33:50.888441 finding duplicate files...
+I 11:34:01.120800 saved 28.2 GiB / 47.65 GiB in 1782826/1927501 duplicate files
+I 11:34:01.122608 waiting for inode scanners...
+I 11:34:12.839065 assigning device inodes...
+I 11:34:12.875520 assigning pipe/socket inodes...
+I 11:34:12.910431 building metadata...
+I 11:34:12.910524 building blocks...
+I 11:34:12.910594 saving names and links...
+I 11:34:12.910691 bloom filter size: 32 KiB
+I 11:34:12.910760 ordering 144675 inodes using nilsimsa similarity...
+I 11:34:12.915555 nilsimsa: depth=20000 (1000), limit=255
+I 11:34:13.052525 updating name and link indices...
+I 11:34:13.276233 pre-sorted index (660176 name, 366179 path lookups) [360.6ms]
+I 11:35:44.039375 144675 inodes ordered [91.13s]
+I 11:35:44.041427 waiting for segmenting/blockifying to finish...
+I 11:37:38.823902 bloom filter reject rate: 96.017% (TPR=0.244%, lookups=4740563665)
+I 11:37:38.823963 segmentation matches: good=454708, bad=6819, total=464247
+I 11:37:38.824005 segmentation collisions: L1=0.008%, L2=0.000% [2233254 hashes]
+I 11:37:38.824038 saving chunks...
+I 11:37:38.860939 saving directories...
+I 11:37:41.318747 waiting for compression to finish...
+I 11:38:56.046809 compressed 47.65 GiB to 430.9 MiB (ratio=0.00883101)
+I 11:38:56.304922 filesystem created without errors [323s]
+⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
+waiting for block compression to finish
+330733 dirs, 0/2440 soft/hard links, 1927501/1927501 files, 0 other
+original size: 47.65 GiB, dedupe: 28.2 GiB (1782826 files), segment: 15.19 GiB
+filesystem: 4.261 GiB in 273 blocks (319178 chunks, 144675/144675 inodes)
+compressed filesystem: 273 blocks/430.9 MiB written [depth: 20000]
+█████████████████████████████████████████████████████████████████████████████▏100% |
+
+real 5m23.030s
+user 78m7.554s
+sys 1m47.968s
+```
So in this comparison, `mkdwarfs` is **more than 6 times faster** than `mksquashfs`,
both in terms of CPU time and wall clock time.
- $ ll perl-install.*fs
- -rw-r--r-- 1 mhx users 447230618 Mar 3 20:28 perl-install.dwarfs
- -rw-r--r-- 1 mhx users 4748902400 Mar 3 20:10 perl-install.squashfs
+```
+$ ll perl-install.*fs
+-rw-r--r-- 1 mhx users 447230618 Mar 3 20:28 perl-install.dwarfs
+-rw-r--r-- 1 mhx users 4748902400 Mar 3 20:10 perl-install.squashfs
+```
In terms of compression ratio, the **DwarFS file system is more than 10 times
smaller than the SquashFS file system**. With DwarFS, the content has been
@@ -351,21 +372,27 @@ the original space**.
Here's another comparison using `lzma` compression instead of `zstd`:
- $ time mksquashfs install perl-install-lzma.squashfs -comp lzma
-
- real 13m42.825s
- user 205m40.851s
- sys 3m29.088s
+```
+$ time mksquashfs install perl-install-lzma.squashfs -comp lzma
- $ time mkdwarfs -i install -o perl-install-lzma.dwarfs -l9
-
- real 3m43.937s
- user 49m45.295s
- sys 1m44.550s
+real 13m42.825s
+user 205m40.851s
+sys 3m29.088s
+```
- $ ll perl-install-lzma.*fs
- -rw-r--r-- 1 mhx users 315482627 Mar 3 21:23 perl-install-lzma.dwarfs
- -rw-r--r-- 1 mhx users 3838406656 Mar 3 20:50 perl-install-lzma.squashfs
+```
+$ time mkdwarfs -i install -o perl-install-lzma.dwarfs -l9
+
+real 3m43.937s
+user 49m45.295s
+sys 1m44.550s
+```
+
+```
+$ ll perl-install-lzma.*fs
+-rw-r--r-- 1 mhx users 315482627 Mar 3 21:23 perl-install-lzma.dwarfs
+-rw-r--r-- 1 mhx users 3838406656 Mar 3 20:50 perl-install-lzma.squashfs
+```
It's immediately obvious that the runs are significantly faster and the
resulting images are significantly smaller. Still, `mkdwarfs` is about
@@ -383,21 +410,27 @@ uses a block size of 128KiB, whereas `mkdwarfs` uses 16MiB blocks by default,
or even 64MiB blocks with `-l9`. When using identical block sizes for both
file systems, the difference, quite expectedly, becomes a lot less dramatic:
- $ time mksquashfs install perl-install-lzma-1M.squashfs -comp lzma -b 1M
-
- real 15m43.319s
- user 139m24.533s
- sys 0m45.132s
+```
+$ time mksquashfs install perl-install-lzma-1M.squashfs -comp lzma -b 1M
- $ time mkdwarfs -i install -o perl-install-lzma-1M.dwarfs -l9 -S20 -B3
-
- real 4m25.973s
- user 52m15.100s
- sys 7m41.889s
+real 15m43.319s
+user 139m24.533s
+sys 0m45.132s
+```
- $ ll perl-install*.*fs
- -rw-r--r-- 1 mhx users 935953866 Mar 13 12:12 perl-install-lzma-1M.dwarfs
- -rw-r--r-- 1 mhx users 3407474688 Mar 3 21:54 perl-install-lzma-1M.squashfs
+```
+$ time mkdwarfs -i install -o perl-install-lzma-1M.dwarfs -l9 -S20 -B3
+
+real 4m25.973s
+user 52m15.100s
+sys 7m41.889s
+```
+
+```
+$ ll perl-install*.*fs
+-rw-r--r-- 1 mhx users 935953866 Mar 13 12:12 perl-install-lzma-1M.dwarfs
+-rw-r--r-- 1 mhx users 3407474688 Mar 3 21:54 perl-install-lzma-1M.squashfs
+```
Even this is *still* not entirely fair, as it uses a feature (`-B3`) that allows
DwarFS to reference file chunks from up to two previous filesystem blocks.
@@ -413,21 +446,25 @@ fast experimentation with different algorithms and options without requiring
a full rebuild of the file system. For example, recompressing the above file
system with the best possible compression (`-l 9`):
- $ time mkdwarfs --recompress -i perl-install.dwarfs -o perl-lzma-re.dwarfs -l9
- I 20:28:03.246534 filesystem rewrittenwithout errors [148.3s]
- ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
- filesystem: 4.261 GiB in 273 blocks (0 chunks, 0 inodes)
- compressed filesystem: 273/273 blocks/372.7 MiB written
- ████████████████████████████████████████████████████████████████████▏100% \
-
- real 2m28.279s
- user 37m8.825s
- sys 0m43.256s
+```
+$ time mkdwarfs --recompress -i perl-install.dwarfs -o perl-lzma-re.dwarfs -l9
+I 20:28:03.246534 filesystem rewrittenwithout errors [148.3s]
+⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
+filesystem: 4.261 GiB in 273 blocks (0 chunks, 0 inodes)
+compressed filesystem: 273/273 blocks/372.7 MiB written
+████████████████████████████████████████████████████████████████████▏100% \
- $ ll perl-*.dwarfs
- -rw-r--r-- 1 mhx users 447230618 Mar 3 20:28 perl-install.dwarfs
- -rw-r--r-- 1 mhx users 390845518 Mar 4 20:28 perl-lzma-re.dwarfs
- -rw-r--r-- 1 mhx users 315482627 Mar 3 21:23 perl-install-lzma.dwarfs
+real 2m28.279s
+user 37m8.825s
+sys 0m43.256s
+```
+
+```
+$ ll perl-*.dwarfs
+-rw-r--r-- 1 mhx users 447230618 Mar 3 20:28 perl-install.dwarfs
+-rw-r--r-- 1 mhx users 390845518 Mar 4 20:28 perl-lzma-re.dwarfs
+-rw-r--r-- 1 mhx users 315482627 Mar 3 21:23 perl-install-lzma.dwarfs
+```
Note that while the recompressed filesystem is smaller than the original image,
it is still a lot bigger than the filesystem we previously build with `-l9`.
@@ -438,22 +475,24 @@ In terms of how fast the file system is when using it, a quick test
I've done is to freshly mount the filesystem created above and run
each of the 1139 `perl` executables to print their version.
- $ hyperfine -c "umount mnt" -p "umount mnt; dwarfs perl-install.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1" -P procs 5 20 -D 5 "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'"
- Benchmark #1: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P5 sh -c '$0 -v >/dev/null'
- Time (mean ± σ): 1.810 s ± 0.013 s [User: 1.847 s, System: 0.623 s]
- Range (min … max): 1.788 s … 1.825 s 10 runs
-
- Benchmark #2: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
- Time (mean ± σ): 1.333 s ± 0.009 s [User: 1.993 s, System: 0.656 s]
- Range (min … max): 1.321 s … 1.354 s 10 runs
-
- Benchmark #3: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P15 sh -c '$0 -v >/dev/null'
- Time (mean ± σ): 1.181 s ± 0.018 s [User: 2.086 s, System: 0.712 s]
- Range (min … max): 1.165 s … 1.214 s 10 runs
-
- Benchmark #4: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
- Time (mean ± σ): 1.149 s ± 0.015 s [User: 2.128 s, System: 0.781 s]
- Range (min … max): 1.136 s … 1.186 s 10 runs
+```
+$ hyperfine -c "umount mnt" -p "umount mnt; dwarfs perl-install.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1" -P procs 5 20 -D 5 "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'"
+Benchmark #1: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P5 sh -c '$0 -v >/dev/null'
+ Time (mean ± σ): 1.810 s ± 0.013 s [User: 1.847 s, System: 0.623 s]
+ Range (min … max): 1.788 s … 1.825 s 10 runs
+
+Benchmark #2: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
+ Time (mean ± σ): 1.333 s ± 0.009 s [User: 1.993 s, System: 0.656 s]
+ Range (min … max): 1.321 s … 1.354 s 10 runs
+
+Benchmark #3: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P15 sh -c '$0 -v >/dev/null'
+ Time (mean ± σ): 1.181 s ± 0.018 s [User: 2.086 s, System: 0.712 s]
+ Range (min … max): 1.165 s … 1.214 s 10 runs
+
+Benchmark #4: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
+ Time (mean ± σ): 1.149 s ± 0.015 s [User: 2.128 s, System: 0.781 s]
+ Range (min … max): 1.136 s … 1.186 s 10 runs
+```
These timings are for *initial* runs on a freshly mounted file system,
running 5, 10, 15 and 20 processes in parallel. 1.1 seconds means that
@@ -462,48 +501,52 @@ it takes only about 1 millisecond per Perl binary.
Following are timings for *subsequent* runs, both on DwarFS (at `mnt`)
and the original XFS (at `install`). DwarFS is around 15% slower here:
- $ hyperfine -P procs 10 20 -D 10 -w1 "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'" "ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'"
- Benchmark #1: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
- Time (mean ± σ): 347.0 ms ± 7.2 ms [User: 1.755 s, System: 0.452 s]
- Range (min … max): 341.3 ms … 365.2 ms 10 runs
-
- Benchmark #2: ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
- Time (mean ± σ): 302.5 ms ± 3.3 ms [User: 1.656 s, System: 0.377 s]
- Range (min … max): 297.1 ms … 308.7 ms 10 runs
-
- Benchmark #3: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
- Time (mean ± σ): 342.2 ms ± 4.1 ms [User: 1.766 s, System: 0.451 s]
- Range (min … max): 336.0 ms … 349.7 ms 10 runs
-
- Benchmark #4: ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
- Time (mean ± σ): 302.0 ms ± 3.0 ms [User: 1.659 s, System: 0.374 s]
- Range (min … max): 297.0 ms … 305.4 ms 10 runs
-
- Summary
- 'ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'' ran
- 1.00 ± 0.01 times faster than 'ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null''
- 1.13 ± 0.02 times faster than 'ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null''
- 1.15 ± 0.03 times faster than 'ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null''
+```
+$ hyperfine -P procs 10 20 -D 10 -w1 "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'" "ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'"
+Benchmark #1: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
+ Time (mean ± σ): 347.0 ms ± 7.2 ms [User: 1.755 s, System: 0.452 s]
+ Range (min … max): 341.3 ms … 365.2 ms 10 runs
+
+Benchmark #2: ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
+ Time (mean ± σ): 302.5 ms ± 3.3 ms [User: 1.656 s, System: 0.377 s]
+ Range (min … max): 297.1 ms … 308.7 ms 10 runs
+
+Benchmark #3: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
+ Time (mean ± σ): 342.2 ms ± 4.1 ms [User: 1.766 s, System: 0.451 s]
+ Range (min … max): 336.0 ms … 349.7 ms 10 runs
+
+Benchmark #4: ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
+ Time (mean ± σ): 302.0 ms ± 3.0 ms [User: 1.659 s, System: 0.374 s]
+ Range (min … max): 297.0 ms … 305.4 ms 10 runs
+
+Summary
+ 'ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'' ran
+ 1.00 ± 0.01 times faster than 'ls -1 install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null''
+ 1.13 ± 0.02 times faster than 'ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null''
+ 1.15 ± 0.03 times faster than 'ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null''
+```
Using the lzma-compressed file system, the metrics for *initial* runs look
considerably worse (about an order of magnitude):
- $ hyperfine -c "umount mnt" -p "umount mnt; dwarfs perl-install-lzma.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1" -P procs 5 20 -D 5 "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'"
- Benchmark #1: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P5 sh -c '$0 -v >/dev/null'
- Time (mean ± σ): 10.660 s ± 0.057 s [User: 1.952 s, System: 0.729 s]
- Range (min … max): 10.615 s … 10.811 s 10 runs
-
- Benchmark #2: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
- Time (mean ± σ): 9.092 s ± 0.021 s [User: 1.979 s, System: 0.680 s]
- Range (min … max): 9.059 s … 9.126 s 10 runs
-
- Benchmark #3: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P15 sh -c '$0 -v >/dev/null'
- Time (mean ± σ): 9.012 s ± 0.188 s [User: 2.077 s, System: 0.702 s]
- Range (min … max): 8.839 s … 9.277 s 10 runs
-
- Benchmark #4: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
- Time (mean ± σ): 9.004 s ± 0.298 s [User: 2.134 s, System: 0.736 s]
- Range (min … max): 8.611 s … 9.555 s 10 runs
+```
+$ hyperfine -c "umount mnt" -p "umount mnt; dwarfs perl-install-lzma.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1" -P procs 5 20 -D 5 "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P{procs} sh -c '\$0 -v >/dev/null'"
+Benchmark #1: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P5 sh -c '$0 -v >/dev/null'
+ Time (mean ± σ): 10.660 s ± 0.057 s [User: 1.952 s, System: 0.729 s]
+ Range (min … max): 10.615 s … 10.811 s 10 runs
+
+Benchmark #2: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P10 sh -c '$0 -v >/dev/null'
+ Time (mean ± σ): 9.092 s ± 0.021 s [User: 1.979 s, System: 0.680 s]
+ Range (min … max): 9.059 s … 9.126 s 10 runs
+
+Benchmark #3: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P15 sh -c '$0 -v >/dev/null'
+ Time (mean ± σ): 9.012 s ± 0.188 s [User: 2.077 s, System: 0.702 s]
+ Range (min … max): 8.839 s … 9.277 s 10 runs
+
+Benchmark #4: ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '$0 -v >/dev/null'
+ Time (mean ± σ): 9.004 s ± 0.298 s [User: 2.134 s, System: 0.736 s]
+ Range (min … max): 8.611 s … 9.555 s 10 runs
+```
So you might want to consider using `zstd` instead of `lzma` if you'd
like to optimize for file system performance. It's also the default
@@ -511,29 +554,33 @@ compression used by `mkdwarfs`.
Now here's a comparison with the SquashFS filesystem:
- $ hyperfine -c 'sudo umount mnt' -p 'umount mnt; dwarfs perl-install.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1' -n dwarfs-zstd "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '\$0 -v >/dev/null'" -p 'sudo umount mnt; sudo mount -t squashfs perl-install.squashfs mnt; sleep 1' -n squashfs-zstd "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '\$0 -v >/dev/null'"
- Benchmark #1: dwarfs-zstd
- Time (mean ± σ): 1.151 s ± 0.015 s [User: 2.147 s, System: 0.769 s]
- Range (min … max): 1.118 s … 1.174 s 10 runs
-
- Benchmark #2: squashfs-zstd
- Time (mean ± σ): 6.733 s ± 0.007 s [User: 3.188 s, System: 17.015 s]
- Range (min … max): 6.721 s … 6.743 s 10 runs
-
- Summary
- 'dwarfs-zstd' ran
- 5.85 ± 0.08 times faster than 'squashfs-zstd'
+```
+$ hyperfine -c 'sudo umount mnt' -p 'umount mnt; dwarfs perl-install.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1' -n dwarfs-zstd "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '\$0 -v >/dev/null'" -p 'sudo umount mnt; sudo mount -t squashfs perl-install.squashfs mnt; sleep 1' -n squashfs-zstd "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '\$0 -v >/dev/null'"
+Benchmark #1: dwarfs-zstd
+ Time (mean ± σ): 1.151 s ± 0.015 s [User: 2.147 s, System: 0.769 s]
+ Range (min … max): 1.118 s … 1.174 s 10 runs
+
+Benchmark #2: squashfs-zstd
+ Time (mean ± σ): 6.733 s ± 0.007 s [User: 3.188 s, System: 17.015 s]
+ Range (min … max): 6.721 s … 6.743 s 10 runs
+
+Summary
+ 'dwarfs-zstd' ran
+ 5.85 ± 0.08 times faster than 'squashfs-zstd'
+```
So DwarFS is almost six times faster than SquashFS. But what's more,
SquashFS also uses significantly more CPU power. However, the numbers
shown above for DwarFS obviously don't include the time spent in the
`dwarfs` process, so I repeated the test outside of hyperfine:
- $ time dwarfs perl-install.dwarfs mnt -o cachesize=1g -o workers=4 -f
-
- real 0m4.569s
- user 0m2.154s
- sys 0m1.846s
+```
+$ time dwarfs perl-install.dwarfs mnt -o cachesize=1g -o workers=4 -f
+
+real 0m4.569s
+user 0m2.154s
+sys 0m1.846s
+```
So in total, DwarFS was using 5.7 seconds of CPU time, whereas
SquashFS was using 20.2 seconds, almost four times as much. Ignore
@@ -546,13 +593,13 @@ used, [Tie::Hash::Indexed](https://github.com/mhx/Tie-Hash-Indexed),
has an XS component that requires a C compiler to build. So this really
accesses a lot of different stuff in the file system:
-* The `perl` executables and its shared libraries
+- The `perl` executables and its shared libraries
-* The Perl modules used for writing the Makefile
+- The Perl modules used for writing the Makefile
-* Perl's C header files used for building the module
+- Perl's C header files used for building the module
-* More Perl modules used for running the tests
+- More Perl modules used for running the tests
I wrote a little script to be able to run multiple builds in parallel:
@@ -574,28 +621,36 @@ The following command will run up to 16 builds in parallel on the 8 core
Xeon CPU, including debug, optimized and threaded versions of all Perl
releases between 5.10.0 and 5.33.3, a total of 624 `perl` installations:
- $ time ls -1 /tmp/perl/install/*/perl-5.??.?/bin/perl5* | sort -t / -k 8 | xargs -d $'\n' -P 16 -n 1 ./build.sh
+```
+$ time ls -1 /tmp/perl/install/*/perl-5.??.?/bin/perl5* | sort -t / -k 8 | xargs -d $'\n' -P 16 -n 1 ./build.sh
+```
Tests were done with a cleanly mounted file system to make sure the caches
were empty. `ccache` was primed to make sure all compiler runs could be
satisfied from the cache. With SquashFS, the timing was:
- real 0m52.385s
- user 8m10.333s
- sys 4m10.056s
+```
+real 0m52.385s
+user 8m10.333s
+sys 4m10.056s
+```
And with DwarFS:
- real 0m50.469s
- user 9m22.597s
- sys 1m18.469s
+```
+real 0m50.469s
+user 9m22.597s
+sys 1m18.469s
+```
So, frankly, not much of a difference, with DwarFS being just a bit faster.
The `dwarfs` process itself used:
- real 0m56.686s
- user 0m18.857s
- sys 0m21.058s
+```
+real 0m56.686s
+user 0m18.857s
+sys 0m21.058s
+```
So again, DwarFS used less raw CPU power overall, but in terms of wallclock
time, the difference is really marginal.
@@ -606,150 +661,168 @@ This test uses slightly less pathological input data: the root filesystem of
a recent Raspberry Pi OS release. This file system also contains device inodes,
so in order to preserve those, we pass `--with-devices` to `mkdwarfs`:
- $ time sudo mkdwarfs -i raspbian -o raspbian.dwarfs --with-devices
- I 21:30:29.812562 scanning raspbian
- I 21:30:29.908984 waiting for background scanners...
- I 21:30:30.217446 assigning directory and link inodes...
- I 21:30:30.221941 finding duplicate files...
- I 21:30:30.288099 saved 31.05 MiB / 1007 MiB in 1617/34582 duplicate files
- I 21:30:30.288143 waiting for inode scanners...
- I 21:30:31.393710 assigning device inodes...
- I 21:30:31.394481 assigning pipe/socket inodes...
- I 21:30:31.395196 building metadata...
- I 21:30:31.395230 building blocks...
- I 21:30:31.395291 saving names and links...
- I 21:30:31.395374 ordering 32965 inodes using nilsimsa similarity...
- I 21:30:31.396254 nilsimsa: depth=20000 (1000), limit=255
- I 21:30:31.407967 pre-sorted index (46431 name, 2206 path lookups) [11.66ms]
- I 21:30:31.410089 updating name and link indices...
- I 21:30:38.178505 32965 inodes ordered [6.783s]
- I 21:30:38.179417 waiting for segmenting/blockifying to finish...
- I 21:31:06.248304 saving chunks...
- I 21:31:06.251998 saving directories...
- I 21:31:06.402559 waiting for compression to finish...
- I 21:31:16.425563 compressed 1007 MiB to 287 MiB (ratio=0.285036)
- I 21:31:16.464772 filesystem created without errors [46.65s]
- ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
- waiting for block compression to finish
- 4435 dirs, 5908/0 soft/hard links, 34582/34582 files, 7 other
- original size: 1007 MiB, dedupe: 31.05 MiB (1617 files), segment: 47.23 MiB
- filesystem: 928.4 MiB in 59 blocks (38890 chunks, 32965/32965 inodes)
- compressed filesystem: 59 blocks/287 MiB written [depth: 20000]
- ████████████████████████████████████████████████████████████████████▏100% |
-
- real 0m46.711s
- user 10m39.038s
- sys 0m8.123s
+```
+$ time sudo mkdwarfs -i raspbian -o raspbian.dwarfs --with-devices
+I 21:30:29.812562 scanning raspbian
+I 21:30:29.908984 waiting for background scanners...
+I 21:30:30.217446 assigning directory and link inodes...
+I 21:30:30.221941 finding duplicate files...
+I 21:30:30.288099 saved 31.05 MiB / 1007 MiB in 1617/34582 duplicate files
+I 21:30:30.288143 waiting for inode scanners...
+I 21:30:31.393710 assigning device inodes...
+I 21:30:31.394481 assigning pipe/socket inodes...
+I 21:30:31.395196 building metadata...
+I 21:30:31.395230 building blocks...
+I 21:30:31.395291 saving names and links...
+I 21:30:31.395374 ordering 32965 inodes using nilsimsa similarity...
+I 21:30:31.396254 nilsimsa: depth=20000 (1000), limit=255
+I 21:30:31.407967 pre-sorted index (46431 name, 2206 path lookups) [11.66ms]
+I 21:30:31.410089 updating name and link indices...
+I 21:30:38.178505 32965 inodes ordered [6.783s]
+I 21:30:38.179417 waiting for segmenting/blockifying to finish...
+I 21:31:06.248304 saving chunks...
+I 21:31:06.251998 saving directories...
+I 21:31:06.402559 waiting for compression to finish...
+I 21:31:16.425563 compressed 1007 MiB to 287 MiB (ratio=0.285036)
+I 21:31:16.464772 filesystem created without errors [46.65s]
+⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
+waiting for block compression to finish
+4435 dirs, 5908/0 soft/hard links, 34582/34582 files, 7 other
+original size: 1007 MiB, dedupe: 31.05 MiB (1617 files), segment: 47.23 MiB
+filesystem: 928.4 MiB in 59 blocks (38890 chunks, 32965/32965 inodes)
+compressed filesystem: 59 blocks/287 MiB written [depth: 20000]
+████████████████████████████████████████████████████████████████████▏100% |
+
+real 0m46.711s
+user 10m39.038s
+sys 0m8.123s
+```
Again, SquashFS uses the same compression options:
- $ time sudo mksquashfs raspbian raspbian.squashfs -comp zstd -Xcompression-level 22
- Parallel mksquashfs: Using 16 processors
- Creating 4.0 filesystem on raspbian.squashfs, block size 131072.
- [===============================================================\] 39232/39232 100%
-
- Exportable Squashfs 4.0 filesystem, zstd compressed, data block size 131072
- compressed data, compressed metadata, compressed fragments,
- compressed xattrs, compressed ids
- duplicates are removed
- Filesystem size 371934.50 Kbytes (363.22 Mbytes)
- 35.98% of uncompressed filesystem size (1033650.60 Kbytes)
- Inode table size 399913 bytes (390.54 Kbytes)
- 26.53% of uncompressed inode table size (1507581 bytes)
- Directory table size 408749 bytes (399.17 Kbytes)
- 42.31% of uncompressed directory table size (966174 bytes)
- Number of duplicate files found 1618
- Number of inodes 44932
- Number of files 34582
- Number of fragments 3290
- Number of symbolic links 5908
- Number of device nodes 7
- Number of fifo nodes 0
- Number of socket nodes 0
- Number of directories 4435
- Number of ids (unique uids + gids) 18
- Number of uids 5
- root (0)
- mhx (1000)
- unknown (103)
- shutdown (6)
- unknown (106)
- Number of gids 15
- root (0)
- unknown (109)
- unknown (42)
- unknown (1000)
- users (100)
- unknown (43)
- tty (5)
- unknown (108)
- unknown (111)
- unknown (110)
- unknown (50)
- mail (12)
- nobody (65534)
- adm (4)
- mem (8)
-
- real 0m50.124s
- user 9m41.708s
- sys 0m1.727s
+```
+$ time sudo mksquashfs raspbian raspbian.squashfs -comp zstd -Xcompression-level 22
+Parallel mksquashfs: Using 16 processors
+Creating 4.0 filesystem on raspbian.squashfs, block size 131072.
+[===============================================================\] 39232/39232 100%
+
+Exportable Squashfs 4.0 filesystem, zstd compressed, data block size 131072
+ compressed data, compressed metadata, compressed fragments,
+ compressed xattrs, compressed ids
+ duplicates are removed
+Filesystem size 371934.50 Kbytes (363.22 Mbytes)
+ 35.98% of uncompressed filesystem size (1033650.60 Kbytes)
+Inode table size 399913 bytes (390.54 Kbytes)
+ 26.53% of uncompressed inode table size (1507581 bytes)
+Directory table size 408749 bytes (399.17 Kbytes)
+ 42.31% of uncompressed directory table size (966174 bytes)
+Number of duplicate files found 1618
+Number of inodes 44932
+Number of files 34582
+Number of fragments 3290
+Number of symbolic links 5908
+Number of device nodes 7
+Number of fifo nodes 0
+Number of socket nodes 0
+Number of directories 4435
+Number of ids (unique uids + gids) 18
+Number of uids 5
+ root (0)
+ mhx (1000)
+ unknown (103)
+ shutdown (6)
+ unknown (106)
+Number of gids 15
+ root (0)
+ unknown (109)
+ unknown (42)
+ unknown (1000)
+ users (100)
+ unknown (43)
+ tty (5)
+ unknown (108)
+ unknown (111)
+ unknown (110)
+ unknown (50)
+ mail (12)
+ nobody (65534)
+ adm (4)
+ mem (8)
+
+real 0m50.124s
+user 9m41.708s
+sys 0m1.727s
+```
The difference in speed is almost negligible. SquashFS is just a bit
slower here. In terms of compression, the difference also isn't huge:
- $ ls -lh raspbian.* *.xz
- -rw-r--r-- 1 mhx users 297M Mar 4 21:32 2020-08-20-raspios-buster-armhf-lite.img.xz
- -rw-r--r-- 1 root root 287M Mar 4 21:31 raspbian.dwarfs
- -rw-r--r-- 1 root root 364M Mar 4 21:33 raspbian.squashfs
+```
+$ ls -lh raspbian.* *.xz
+-rw-r--r-- 1 mhx users 297M Mar 4 21:32 2020-08-20-raspios-buster-armhf-lite.img.xz
+-rw-r--r-- 1 root root 287M Mar 4 21:31 raspbian.dwarfs
+-rw-r--r-- 1 root root 364M Mar 4 21:33 raspbian.squashfs
+```
Interestingly, `xz` actually can't compress the whole original image
better than DwarFS.
We can even again try to increase the DwarFS compression level:
- $ time sudo mkdwarfs -i raspbian -o raspbian-9.dwarfs --with-devices -l9
-
- real 0m54.161s
- user 8m40.109s
- sys 0m7.101s
+```
+$ time sudo mkdwarfs -i raspbian -o raspbian-9.dwarfs --with-devices -l9
+
+real 0m54.161s
+user 8m40.109s
+sys 0m7.101s
+```
Now that actually gets the DwarFS image size well below that of the
`xz` archive:
- $ ls -lh raspbian-9.dwarfs *.xz
- -rw-r--r-- 1 root root 244M Mar 4 21:36 raspbian-9.dwarfs
- -rw-r--r-- 1 mhx users 297M Mar 4 21:32 2020-08-20-raspios-buster-armhf-lite.img.xz
+```
+$ ls -lh raspbian-9.dwarfs *.xz
+-rw-r--r-- 1 root root 244M Mar 4 21:36 raspbian-9.dwarfs
+-rw-r--r-- 1 mhx users 297M Mar 4 21:32 2020-08-20-raspios-buster-armhf-lite.img.xz
+```
Even if you actually build a tarball and compress that (instead of
compressing the EXT4 file system itself), `xz` isn't quite able to
match the DwarFS image size:
- $ time sudo tar cf - raspbian | xz -9 -vT 0 >raspbian.tar.xz
- 100 % 246.9 MiB / 1,037.2 MiB = 0.238 13 MiB/s 1:18
-
- real 1m18.226s
- user 6m35.381s
- sys 0m2.205s
+```
+$ time sudo tar cf - raspbian | xz -9 -vT 0 >raspbian.tar.xz
+ 100 % 246.9 MiB / 1,037.2 MiB = 0.238 13 MiB/s 1:18
- $ ls -lh raspbian.tar.xz
- -rw-r--r-- 1 mhx users 247M Mar 4 21:40 raspbian.tar.xz
+real 1m18.226s
+user 6m35.381s
+sys 0m2.205s
+```
+
+```
+$ ls -lh raspbian.tar.xz
+-rw-r--r-- 1 mhx users 247M Mar 4 21:40 raspbian.tar.xz
+```
DwarFS also comes with the [dwarfsextract](doc/dwarfsextract.md) tool
that allows extraction of a filesystem image without the FUSE driver.
So here's a comparison of the extraction speed:
- $ time sudo tar xf raspbian.tar.xz -C out1
-
- real 0m12.846s
- user 0m12.313s
- sys 0m1.616s
+```
+$ time sudo tar xf raspbian.tar.xz -C out1
- $ time sudo dwarfsextract -i raspbian-9.dwarfs -o out2
-
- real 0m3.825s
- user 0m13.234s
- sys 0m1.382s
+real 0m12.846s
+user 0m12.313s
+sys 0m1.616s
+```
+
+```
+$ time sudo dwarfsextract -i raspbian-9.dwarfs -o out2
+
+real 0m3.825s
+user 0m13.234s
+sys 0m1.382s
+```
So `dwarfsextract` is almost 4 times faster thanks to using multiple
worker threads for decompression. It's writing about 300 MiB/s in this
@@ -759,14 +832,18 @@ Another nice feature of `dwarfsextract` is that it allows you to directly
output data in an archive format, so you could create a tarball from
your image without extracting the files to disk:
- $ dwarfsextract -i raspbian-9.dwarfs -f ustar | xz -9 -T0 >raspbian2.tar.xz
+```
+$ dwarfsextract -i raspbian-9.dwarfs -f ustar | xz -9 -T0 >raspbian2.tar.xz
+```
This has the interesting side-effect that the resulting tarball will
likely be smaller than the one built straight from the directory:
- $ ls -lh raspbian*.tar.xz
- -rw-r--r-- 1 mhx users 247M Mar 4 21:40 raspbian.tar.xz
- -rw-r--r-- 1 mhx users 240M Mar 4 23:52 raspbian2.tar.xz
+```
+$ ls -lh raspbian*.tar.xz
+-rw-r--r-- 1 mhx users 247M Mar 4 21:40 raspbian.tar.xz
+-rw-r--r-- 1 mhx users 240M Mar 4 23:52 raspbian2.tar.xz
+```
That's because `dwarfsextract` writes files in inode-order, and by
default inodes are ordered by similarity for the best possible
@@ -784,42 +861,48 @@ When I first read about `lrzip`, I was pretty certain it would easily
beat DwarFS. So let's take a look. `lrzip` operates on a single file,
so it's necessary to first build a tarball:
- $ time tar cf perl-install.tar install
-
- real 2m9.568s
- user 0m3.757s
- sys 0m26.623s
+```
+$ time tar cf perl-install.tar install
+
+real 2m9.568s
+user 0m3.757s
+sys 0m26.623s
+```
Now we can run `lrzip`:
- $ time lrzip -vL9 -o perl-install.tar.lrzip perl-install.tar
- The following options are in effect for this COMPRESSION.
- Threading is ENABLED. Number of CPUs detected: 16
- Detected 67106172928 bytes ram
- Compression level 9
- Nice Value: 19
- Show Progress
- Verbose
- Output Filename Specified: perl-install.tar.lrzip
- Temporary Directory set as: ./
- Compression mode is: LZMA. LZO Compressibility testing enabled
- Heuristically Computed Compression Window: 426 = 42600MB
- File size: 52615639040
- Will take 2 passes
- Beginning rzip pre-processing phase
- Beginning rzip pre-processing phase
- perl-install.tar - Compression Ratio: 100.378. Average Compression Speed: 14.536MB/s.
- Total time: 00:57:32.47
-
- real 57m32.472s
- user 81m44.104s
- sys 4m50.221s
+```
+$ time lrzip -vL9 -o perl-install.tar.lrzip perl-install.tar
+The following options are in effect for this COMPRESSION.
+Threading is ENABLED. Number of CPUs detected: 16
+Detected 67106172928 bytes ram
+Compression level 9
+Nice Value: 19
+Show Progress
+Verbose
+Output Filename Specified: perl-install.tar.lrzip
+Temporary Directory set as: ./
+Compression mode is: LZMA. LZO Compressibility testing enabled
+Heuristically Computed Compression Window: 426 = 42600MB
+File size: 52615639040
+Will take 2 passes
+Beginning rzip pre-processing phase
+Beginning rzip pre-processing phase
+perl-install.tar - Compression Ratio: 100.378. Average Compression Speed: 14.536MB/s.
+Total time: 00:57:32.47
+
+real 57m32.472s
+user 81m44.104s
+sys 4m50.221s
+```
That definitely took a while. This is about an order of magnitude
slower than `mkdwarfs` and it barely makes use of the 8 cores.
- $ ll -h perl-install.tar.lrzip
- -rw-r--r-- 1 mhx users 500M Mar 6 21:16 perl-install.tar.lrzip
+```
+$ ll -h perl-install.tar.lrzip
+-rw-r--r-- 1 mhx users 500M Mar 6 21:16 perl-install.tar.lrzip
+```
This is a surprisingly disappointing result. The archive is 65% larger
than a DwarFS image at `-l9` that takes less than 4 minutes to build.
@@ -828,16 +911,20 @@ unpacking the archive first.
That being said, it *is* better than just using `xz` on the tarball:
- $ time xz -T0 -v9 -c perl-install.tar >perl-install.tar.xz
- perl-install.tar (1/1)
- 100 % 4,317.0 MiB / 49.0 GiB = 0.086 24 MiB/s 34:55
-
- real 34m55.450s
- user 543m50.810s
- sys 0m26.533s
+```
+$ time xz -T0 -v9 -c perl-install.tar >perl-install.tar.xz
+perl-install.tar (1/1)
+ 100 % 4,317.0 MiB / 49.0 GiB = 0.086 24 MiB/s 34:55
- $ ll perl-install.tar.xz -h
- -rw-r--r-- 1 mhx users 4.3G Mar 6 22:59 perl-install.tar.xz
+real 34m55.450s
+user 543m50.810s
+sys 0m26.533s
+```
+
+```
+$ ll perl-install.tar.xz -h
+-rw-r--r-- 1 mhx users 4.3G Mar 6 22:59 perl-install.tar.xz
+```
### With zpaq
@@ -850,42 +937,49 @@ can be used.
Anyway, how does it fare in terms of speed and compression performance?
- $ time zpaq a perl-install.zpaq install -m5
+```
+$ time zpaq a perl-install.zpaq install -m5
+```
After a few million lines of output that (I think) cannot be turned off:
- 2258234 +added, 0 -removed.
-
- 0.000000 + (51161.953159 -> 8932.000297 -> 490.227707) = 490.227707 MB
- 2828.082 seconds (all OK)
-
- real 47m8.104s
- user 714m44.286s
- sys 3m6.751s
+```
+2258234 +added, 0 -removed.
+
+0.000000 + (51161.953159 -> 8932.000297 -> 490.227707) = 490.227707 MB
+2828.082 seconds (all OK)
+
+real 47m8.104s
+user 714m44.286s
+sys 3m6.751s
+```
So it's an order of magnitude slower than `mkdwarfs` and uses 14 times
as much CPU resources as `mkdwarfs -l9`. The resulting archive it pretty
close in size to the default configuration DwarFS image, but it's more
than 50% bigger than the image produced by `mkdwarfs -l9`.
- $ ll perl-install*.*
- -rw-r--r-- 1 mhx users 490227707 Mar 7 01:38 perl-install.zpaq
- -rw-r--r-- 1 mhx users 315482627 Mar 3 21:23 perl-install-l9.dwarfs
- -rw-r--r-- 1 mhx users 447230618 Mar 3 20:28 perl-install.dwarfs
+```
+$ ll perl-install*.*
+-rw-r--r-- 1 mhx users 490227707 Mar 7 01:38 perl-install.zpaq
+-rw-r--r-- 1 mhx users 315482627 Mar 3 21:23 perl-install-l9.dwarfs
+-rw-r--r-- 1 mhx users 447230618 Mar 3 20:28 perl-install.dwarfs
+```
What's *really* surprising is how slow it is to extract the `zpaq`
archive again:
- $ time zpaq x perl-install.zpaq
- 2798.097 seconds (all OK)
-
- real 46m38.117s
- user 711m18.734s
- sys 3m47.876s
+```
+$ time zpaq x perl-install.zpaq
+2798.097 seconds (all OK)
+
+real 46m38.117s
+user 711m18.734s
+sys 3m47.876s
+```
That's 700 times slower than extracting the DwarFS image.
-
### With wimlib
[wimlib](https://wimlib.net/) is a really interesting project that is
@@ -896,21 +990,25 @@ quite a rich set of features, so it's definitely worth taking a look at.
I first tried `wimcapture` on the perl dataset:
- $ time wimcapture --unix-data --solid --solid-chunk-size=16M install perl-install.wim
- Scanning "install"
- 47 GiB scanned (1927501 files, 330733 directories)
- Using LZMS compression with 16 threads
- Archiving file data: 19 GiB of 19 GiB (100%) done
-
- real 15m23.310s
- user 174m29.274s
- sys 0m42.921s
+```
+$ time wimcapture --unix-data --solid --solid-chunk-size=16M install perl-install.wim
+Scanning "install"
+47 GiB scanned (1927501 files, 330733 directories)
+Using LZMS compression with 16 threads
+Archiving file data: 19 GiB of 19 GiB (100%) done
- $ ll perl-install.*
- -rw-r--r-- 1 mhx users 447230618 Mar 3 20:28 perl-install.dwarfs
- -rw-r--r-- 1 mhx users 315482627 Mar 3 21:23 perl-install-l9.dwarfs
- -rw-r--r-- 1 mhx users 4748902400 Mar 3 20:10 perl-install.squashfs
- -rw-r--r-- 1 mhx users 1016981520 Mar 6 21:12 perl-install.wim
+real 15m23.310s
+user 174m29.274s
+sys 0m42.921s
+```
+
+```
+$ ll perl-install.*
+-rw-r--r-- 1 mhx users 447230618 Mar 3 20:28 perl-install.dwarfs
+-rw-r--r-- 1 mhx users 315482627 Mar 3 21:23 perl-install-l9.dwarfs
+-rw-r--r-- 1 mhx users 4748902400 Mar 3 20:10 perl-install.squashfs
+-rw-r--r-- 1 mhx users 1016981520 Mar 6 21:12 perl-install.wim
+```
So wimlib is definitely much better than squashfs, in terms of both
compression ratio and speed. DwarFS is however about 3 times faster to
@@ -921,57 +1019,67 @@ When switching to LZMA compression, the DwarFS file system is more than
What's a bit surprising is that mounting a *wim* file takes quite a bit
of time:
- $ time wimmount perl-install.wim mnt
- [WARNING] Mounting a WIM file containing solid-compressed data; file access may be slow.
-
- real 0m2.038s
- user 0m1.764s
- sys 0m0.242s
+```
+$ time wimmount perl-install.wim mnt
+[WARNING] Mounting a WIM file containing solid-compressed data; file access may be slow.
+
+real 0m2.038s
+user 0m1.764s
+sys 0m0.242s
+```
Mounting the DwarFS image takes almost no time in comparison:
- $ time git/github/dwarfs/build-clang-11/dwarfs perl-install-default.dwarfs mnt
- I 00:23:39.238182 dwarfs (v0.4.0, fuse version 35)
-
- real 0m0.003s
- user 0m0.003s
- sys 0m0.000s
+```
+$ time git/github/dwarfs/build-clang-11/dwarfs perl-install-default.dwarfs mnt
+I 00:23:39.238182 dwarfs (v0.4.0, fuse version 35)
+
+real 0m0.003s
+user 0m0.003s
+sys 0m0.000s
+```
That's just because it immediately forks into background by default and
initializes the file system in the background. However, even when
running it in the foreground, initializing the file system takes only
about 60 milliseconds:
- $ dwarfs perl-install.dwarfs mnt -f
- I 00:25:03.186005 dwarfs (v0.4.0, fuse version 35)
- I 00:25:03.248061 file system initialized [60.95ms]
+```
+$ dwarfs perl-install.dwarfs mnt -f
+I 00:25:03.186005 dwarfs (v0.4.0, fuse version 35)
+I 00:25:03.248061 file system initialized [60.95ms]
+```
If you actually build the DwarFS file system with uncompressed metadata,
mounting is basically instantaneous:
- $ dwarfs perl-install-meta.dwarfs mnt -f
- I 00:27:52.667026 dwarfs (v0.4.0, fuse version 35)
- I 00:27:52.671066 file system initialized [2.879ms]
+```
+$ dwarfs perl-install-meta.dwarfs mnt -f
+I 00:27:52.667026 dwarfs (v0.4.0, fuse version 35)
+I 00:27:52.671066 file system initialized [2.879ms]
+```
I've tried running the benchmark where all 1139 `perl` executables
print their version with the wimlib image, but after about 10 minutes,
it still hadn't finished the first run (with the DwarFS image, one run
took slightly more than 2 seconds). I then tried the following instead:
- $ ls -1 /tmp/perl/install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P1 sh -c 'time $0 -v >/dev/null' 2>&1 | grep ^real
- real 0m0.802s
- real 0m0.652s
- real 0m1.677s
- real 0m1.973s
- real 0m1.435s
- real 0m1.879s
- real 0m2.003s
- real 0m1.695s
- real 0m2.343s
- real 0m1.899s
- real 0m1.809s
- real 0m1.790s
- real 0m2.115s
+```
+$ ls -1 /tmp/perl/install/*/*/bin/perl5* | xargs -d $'\n' -n1 -P1 sh -c 'time $0 -v >/dev/null' 2>&1 | grep ^real
+real 0m0.802s
+real 0m0.652s
+real 0m1.677s
+real 0m1.973s
+real 0m1.435s
+real 0m1.879s
+real 0m2.003s
+real 0m1.695s
+real 0m2.343s
+real 0m1.899s
+real 0m1.809s
+real 0m1.790s
+real 0m2.115s
+```
Judging from that, it would have probably taken about half an hour
for a single run, which makes at least the `--solid` wim image pretty
@@ -982,47 +1090,54 @@ that DwarFS actually organizes data internally. However, judging by the
warning when mounting a solid image, it's probably not ideal when using
the image as a mounted file system. So I tried again without `--solid`:
- $ time wimcapture --unix-data install perl-install-nonsolid.wim
- Scanning "install"
- 47 GiB scanned (1927501 files, 330733 directories)
- Using LZX compression with 16 threads
- Archiving file data: 19 GiB of 19 GiB (100%) done
-
- real 8m39.034s
- user 64m58.575s
- sys 0m32.003s
+```
+$ time wimcapture --unix-data install perl-install-nonsolid.wim
+Scanning "install"
+47 GiB scanned (1927501 files, 330733 directories)
+Using LZX compression with 16 threads
+Archiving file data: 19 GiB of 19 GiB (100%) done
+
+real 8m39.034s
+user 64m58.575s
+sys 0m32.003s
+```
This is still more than 3 minutes slower than `mkdwarfs`. However, it
yields an image that's almost 10 times the size of the DwarFS image
and comparable in size to the SquashFS image:
- $ ll perl-install-nonsolid.wim -h
- -rw-r--r-- 1 mhx users 4.6G Mar 6 23:24 perl-install-nonsolid.wim
+```
+$ ll perl-install-nonsolid.wim -h
+-rw-r--r-- 1 mhx users 4.6G Mar 6 23:24 perl-install-nonsolid.wim
+```
This *still* takes surprisingly long to mount:
- $ time wimmount perl-install-nonsolid.wim mnt
-
- real 0m1.603s
- user 0m1.327s
- sys 0m0.275s
+```
+$ time wimmount perl-install-nonsolid.wim mnt
+
+real 0m1.603s
+user 0m1.327s
+sys 0m0.275s
+```
However, it's really usable as a file system, even though it's about
4-5 times slower than the DwarFS image:
- $ hyperfine -c 'umount mnt' -p 'umount mnt; dwarfs perl-install.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1' -n dwarfs "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '\$0 -v >/dev/null'" -p 'umount mnt; wimmount perl-install-nonsolid.wim mnt; sleep 1' -n wimlib "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '\$0 -v >/dev/null'"
- Benchmark #1: dwarfs
- Time (mean ± σ): 1.149 s ± 0.019 s [User: 2.147 s, System: 0.739 s]
- Range (min … max): 1.122 s … 1.187 s 10 runs
-
- Benchmark #2: wimlib
- Time (mean ± σ): 7.542 s ± 0.069 s [User: 2.787 s, System: 0.694 s]
- Range (min … max): 7.490 s … 7.732 s 10 runs
-
- Summary
- 'dwarfs' ran
- 6.56 ± 0.12 times faster than 'wimlib'
+```
+$ hyperfine -c 'umount mnt' -p 'umount mnt; dwarfs perl-install.dwarfs mnt -o cachesize=1g -o workers=4; sleep 1' -n dwarfs "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '\$0 -v >/dev/null'" -p 'umount mnt; wimmount perl-install-nonsolid.wim mnt; sleep 1' -n wimlib "ls -1 mnt/*/*/bin/perl5* | xargs -d $'\n' -n1 -P20 sh -c '\$0 -v >/dev/null'"
+Benchmark #1: dwarfs
+ Time (mean ± σ): 1.149 s ± 0.019 s [User: 2.147 s, System: 0.739 s]
+ Range (min … max): 1.122 s … 1.187 s 10 runs
+Benchmark #2: wimlib
+ Time (mean ± σ): 7.542 s ± 0.069 s [User: 2.787 s, System: 0.694 s]
+ Range (min … max): 7.490 s … 7.732 s 10 runs
+
+Summary
+ 'dwarfs' ran
+ 6.56 ± 0.12 times faster than 'wimlib'
+```
### With Cromfs
@@ -1035,21 +1150,23 @@ Here's a run on the Perl dataset, with the block size set to 16 MiB to
match the default of DwarFS, and with additional options suggested to
speed up compression:
- $ time mkcromfs -f 16777216 -qq -e -r100000 install perl-install.cromfs
- Writing perl-install.cromfs...
- mkcromfs: Automatically enabling --24bitblocknums because it seems possible for this filesystem.
- Root pseudo file is 108 bytes
- Inotab spans 0x7f3a18259000..0x7f3a1bfffb9c
- Root inode spans 0x7f3a205d2948..0x7f3a205d294c
- Beginning task for Files and directories: Finding identical blocks
- 2163608 reuse opportunities found. 561362 unique blocks. Block table will be 79.4% smaller than without the index search.
- Beginning task for Files and directories: Blockifying
- Blockifying: 0.04% (140017/2724970) idx(siz=80423,del=0) rawin(20.97 MB)rawout(20.97 MB)diff(1956 bytes)
- Termination signalled, cleaning up temporaries
-
- real 29m9.634s
- user 201m37.816s
- sys 2m15.005s
+```
+$ time mkcromfs -f 16777216 -qq -e -r100000 install perl-install.cromfs
+Writing perl-install.cromfs...
+mkcromfs: Automatically enabling --24bitblocknums because it seems possible for this filesystem.
+Root pseudo file is 108 bytes
+Inotab spans 0x7f3a18259000..0x7f3a1bfffb9c
+Root inode spans 0x7f3a205d2948..0x7f3a205d294c
+Beginning task for Files and directories: Finding identical blocks
+2163608 reuse opportunities found. 561362 unique blocks. Block table will be 79.4% smaller than without the index search.
+Beginning task for Files and directories: Blockifying
+Blockifying: 0.04% (140017/2724970) idx(siz=80423,del=0) rawin(20.97 MB)rawout(20.97 MB)diff(1956 bytes)
+Termination signalled, cleaning up temporaries
+
+real 29m9.634s
+user 201m37.816s
+sys 2m15.005s
+```
So it processed 21 MiB out of 48 GiB in half an hour, using almost
twice as much CPU resources as DwarFS for the *whole* file system.
@@ -1062,80 +1179,86 @@ I then tried once more with a smaller version of the Perl dataset.
This only has 20 versions (instead of 1139) of Perl, and obviously
a lot less redundancy:
- $ time mkcromfs -f 16777216 -qq -e -r100000 install-small perl-install.cromfs
- Writing perl-install.cromfs...
- mkcromfs: Automatically enabling --16bitblocknums because it seems possible for this filesystem.
- Root pseudo file is 108 bytes
- Inotab spans 0x7f00e0774000..0x7f00e08410a8
- Root inode spans 0x7f00b40048f8..0x7f00b40048fc
- Beginning task for Files and directories: Finding identical blocks
- 25362 reuse opportunities found. 9815 unique blocks. Block table will be 72.1% smaller than without the index search.
- Beginning task for Files and directories: Blockifying
- Compressing raw rootdir inode (28 bytes)z=982370,del=2) rawin(641.56 MB)rawout(252.72 MB)diff(388.84 MB)
- compressed into 35 bytes
- INOTAB pseudo file is 839.85 kB
- Inotab inode spans 0x7f00bc036ed8..0x7f00bc036ef4
- Beginning task for INOTAB: Finding identical blocks
- 0 reuse opportunities found. 13 unique blocks. Block table will be 0.0% smaller than without the index search.
- Beginning task for INOTAB: Blockifying
- mkcromfs: Automatically enabling --packedblocks because it is possible for this filesystem.
- Compressing raw inotab inode (52 bytes)
- compressed into 58 bytes
- Compressing 9828 block records (4 bytes each, total 39312 bytes)
- compressed into 15890 bytes
- Compressing and writing 16 fblocks...
-
- 16 fblocks were written: 35.31 MB = 13.90 % of 254.01 MB
- Filesystem size: 35.33 MB = 5.50 % of original 642.22 MB
- End
-
- real 27m38.833s
- user 277m36.208s
- sys 11m36.945s
+```
+$ time mkcromfs -f 16777216 -qq -e -r100000 install-small perl-install.cromfs
+Writing perl-install.cromfs...
+mkcromfs: Automatically enabling --16bitblocknums because it seems possible for this filesystem.
+Root pseudo file is 108 bytes
+Inotab spans 0x7f00e0774000..0x7f00e08410a8
+Root inode spans 0x7f00b40048f8..0x7f00b40048fc
+Beginning task for Files and directories: Finding identical blocks
+25362 reuse opportunities found. 9815 unique blocks. Block table will be 72.1% smaller than without the index search.
+Beginning task for Files and directories: Blockifying
+Compressing raw rootdir inode (28 bytes)z=982370,del=2) rawin(641.56 MB)rawout(252.72 MB)diff(388.84 MB)
+ compressed into 35 bytes
+INOTAB pseudo file is 839.85 kB
+Inotab inode spans 0x7f00bc036ed8..0x7f00bc036ef4
+Beginning task for INOTAB: Finding identical blocks
+0 reuse opportunities found. 13 unique blocks. Block table will be 0.0% smaller than without the index search.
+Beginning task for INOTAB: Blockifying
+mkcromfs: Automatically enabling --packedblocks because it is possible for this filesystem.
+Compressing raw inotab inode (52 bytes)
+ compressed into 58 bytes
+Compressing 9828 block records (4 bytes each, total 39312 bytes)
+ compressed into 15890 bytes
+Compressing and writing 16 fblocks...
+
+16 fblocks were written: 35.31 MB = 13.90 % of 254.01 MB
+Filesystem size: 35.33 MB = 5.50 % of original 642.22 MB
+End
+
+real 27m38.833s
+user 277m36.208s
+sys 11m36.945s
+```
And repeating the same task with `mkdwarfs`:
- $ time mkdwarfs -i install-small -o perl-install-small.dwarfs
- 21:13:38.131724 scanning install-small
- 21:13:38.320139 waiting for background scanners...
- 21:13:38.727024 assigning directory and link inodes...
- 21:13:38.731807 finding duplicate files...
- 21:13:38.832524 saved 267.8 MiB / 611.8 MiB in 22842/26401 duplicate files
- 21:13:38.832598 waiting for inode scanners...
- 21:13:39.619963 assigning device inodes...
- 21:13:39.620855 assigning pipe/socket inodes...
- 21:13:39.621356 building metadata...
- 21:13:39.621453 building blocks...
- 21:13:39.621472 saving names and links...
- 21:13:39.621655 ordering 3559 inodes using nilsimsa similarity...
- 21:13:39.622031 nilsimsa: depth=20000, limit=255
- 21:13:39.629206 updating name and link indices...
- 21:13:39.630142 pre-sorted index (3360 name, 2127 path lookups) [8.014ms]
- 21:13:39.752051 3559 inodes ordered [130.3ms]
- 21:13:39.752101 waiting for segmenting/blockifying to finish...
- 21:13:53.250951 saving chunks...
- 21:13:53.251581 saving directories...
- 21:13:53.303862 waiting for compression to finish...
- 21:14:11.073273 compressed 611.8 MiB to 24.01 MiB (ratio=0.0392411)
- 21:14:11.091099 filesystem created without errors [32.96s]
- ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
- waiting for block compression to finish
- 3334 dirs, 0/0 soft/hard links, 26401/26401 files, 0 other
- original size: 611.8 MiB, dedupe: 267.8 MiB (22842 files), segment: 121.5 MiB
- filesystem: 222.5 MiB in 14 blocks (7177 chunks, 3559/3559 inodes)
- compressed filesystem: 14 blocks/24.01 MiB written
- ██████████████████████████████████████████████████████████████████████▏100% \
-
- real 0m33.007s
- user 3m43.324s
- sys 0m4.015s
+```
+$ time mkdwarfs -i install-small -o perl-install-small.dwarfs
+21:13:38.131724 scanning install-small
+21:13:38.320139 waiting for background scanners...
+21:13:38.727024 assigning directory and link inodes...
+21:13:38.731807 finding duplicate files...
+21:13:38.832524 saved 267.8 MiB / 611.8 MiB in 22842/26401 duplicate files
+21:13:38.832598 waiting for inode scanners...
+21:13:39.619963 assigning device inodes...
+21:13:39.620855 assigning pipe/socket inodes...
+21:13:39.621356 building metadata...
+21:13:39.621453 building blocks...
+21:13:39.621472 saving names and links...
+21:13:39.621655 ordering 3559 inodes using nilsimsa similarity...
+21:13:39.622031 nilsimsa: depth=20000, limit=255
+21:13:39.629206 updating name and link indices...
+21:13:39.630142 pre-sorted index (3360 name, 2127 path lookups) [8.014ms]
+21:13:39.752051 3559 inodes ordered [130.3ms]
+21:13:39.752101 waiting for segmenting/blockifying to finish...
+21:13:53.250951 saving chunks...
+21:13:53.251581 saving directories...
+21:13:53.303862 waiting for compression to finish...
+21:14:11.073273 compressed 611.8 MiB to 24.01 MiB (ratio=0.0392411)
+21:14:11.091099 filesystem created without errors [32.96s]
+⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
+waiting for block compression to finish
+3334 dirs, 0/0 soft/hard links, 26401/26401 files, 0 other
+original size: 611.8 MiB, dedupe: 267.8 MiB (22842 files), segment: 121.5 MiB
+filesystem: 222.5 MiB in 14 blocks (7177 chunks, 3559/3559 inodes)
+compressed filesystem: 14 blocks/24.01 MiB written
+██████████████████████████████████████████████████████████████████████▏100% \
+
+real 0m33.007s
+user 3m43.324s
+sys 0m4.015s
+```
So `mkdwarfs` is about 50 times faster than `mkcromfs` and uses 75 times
less CPU resources. At the same time, the DwarFS file system is 30% smaller:
- $ ls -l perl-install-small.*fs
- -rw-r--r-- 1 mhx users 35328512 Dec 8 14:25 perl-install-small.cromfs
- -rw-r--r-- 1 mhx users 25175016 Dec 10 21:14 perl-install-small.dwarfs
+```
+$ ls -l perl-install-small.*fs
+-rw-r--r-- 1 mhx users 35328512 Dec 8 14:25 perl-install-small.cromfs
+-rw-r--r-- 1 mhx users 25175016 Dec 10 21:14 perl-install-small.dwarfs
+```
I noticed that the `blockifying` step that took ages for the full dataset
with `mkcromfs` ran substantially faster (in terms of MiB/second) on the
@@ -1145,45 +1268,49 @@ behaviour that's slowing down `mkcromfs`.
In order to be completely fair, I also ran `mkdwarfs` with `-l 9` to enable
LZMA compression (which is what `mkcromfs` uses by default):
- $ time mkdwarfs -i install-small -o perl-install-small-l9.dwarfs -l 9
- 21:16:21.874975 scanning install-small
- 21:16:22.092201 waiting for background scanners...
- 21:16:22.489470 assigning directory and link inodes...
- 21:16:22.495216 finding duplicate files...
- 21:16:22.611221 saved 267.8 MiB / 611.8 MiB in 22842/26401 duplicate files
- 21:16:22.611314 waiting for inode scanners...
- 21:16:23.394332 assigning device inodes...
- 21:16:23.395184 assigning pipe/socket inodes...
- 21:16:23.395616 building metadata...
- 21:16:23.395676 building blocks...
- 21:16:23.395685 saving names and links...
- 21:16:23.395830 ordering 3559 inodes using nilsimsa similarity...
- 21:16:23.396097 nilsimsa: depth=50000, limit=255
- 21:16:23.401042 updating name and link indices...
- 21:16:23.403127 pre-sorted index (3360 name, 2127 path lookups) [6.936ms]
- 21:16:23.524914 3559 inodes ordered [129ms]
- 21:16:23.525006 waiting for segmenting/blockifying to finish...
- 21:16:33.865023 saving chunks...
- 21:16:33.865883 saving directories...
- 21:16:33.900140 waiting for compression to finish...
- 21:17:10.505779 compressed 611.8 MiB to 17.44 MiB (ratio=0.0284969)
- 21:17:10.526171 filesystem created without errors [48.65s]
- ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
- waiting for block compression to finish
- 3334 dirs, 0/0 soft/hard links, 26401/26401 files, 0 other
- original size: 611.8 MiB, dedupe: 267.8 MiB (22842 files), segment: 122.2 MiB
- filesystem: 221.8 MiB in 4 blocks (7304 chunks, 3559/3559 inodes)
- compressed filesystem: 4 blocks/17.44 MiB written
- ██████████████████████████████████████████████████████████████████████▏100% /
-
- real 0m48.683s
- user 2m24.905s
- sys 0m3.292s
+```
+$ time mkdwarfs -i install-small -o perl-install-small-l9.dwarfs -l 9
+21:16:21.874975 scanning install-small
+21:16:22.092201 waiting for background scanners...
+21:16:22.489470 assigning directory and link inodes...
+21:16:22.495216 finding duplicate files...
+21:16:22.611221 saved 267.8 MiB / 611.8 MiB in 22842/26401 duplicate files
+21:16:22.611314 waiting for inode scanners...
+21:16:23.394332 assigning device inodes...
+21:16:23.395184 assigning pipe/socket inodes...
+21:16:23.395616 building metadata...
+21:16:23.395676 building blocks...
+21:16:23.395685 saving names and links...
+21:16:23.395830 ordering 3559 inodes using nilsimsa similarity...
+21:16:23.396097 nilsimsa: depth=50000, limit=255
+21:16:23.401042 updating name and link indices...
+21:16:23.403127 pre-sorted index (3360 name, 2127 path lookups) [6.936ms]
+21:16:23.524914 3559 inodes ordered [129ms]
+21:16:23.525006 waiting for segmenting/blockifying to finish...
+21:16:33.865023 saving chunks...
+21:16:33.865883 saving directories...
+21:16:33.900140 waiting for compression to finish...
+21:17:10.505779 compressed 611.8 MiB to 17.44 MiB (ratio=0.0284969)
+21:17:10.526171 filesystem created without errors [48.65s]
+⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
+waiting for block compression to finish
+3334 dirs, 0/0 soft/hard links, 26401/26401 files, 0 other
+original size: 611.8 MiB, dedupe: 267.8 MiB (22842 files), segment: 122.2 MiB
+filesystem: 221.8 MiB in 4 blocks (7304 chunks, 3559/3559 inodes)
+compressed filesystem: 4 blocks/17.44 MiB written
+██████████████████████████████████████████████████████████████████████▏100% /
- $ ls -l perl-install-small*.*fs
- -rw-r--r-- 1 mhx users 18282075 Dec 10 21:17 perl-install-small-l9.dwarfs
- -rw-r--r-- 1 mhx users 35328512 Dec 8 14:25 perl-install-small.cromfs
- -rw-r--r-- 1 mhx users 25175016 Dec 10 21:14 perl-install-small.dwarfs
+real 0m48.683s
+user 2m24.905s
+sys 0m3.292s
+```
+
+```
+$ ls -l perl-install-small*.*fs
+-rw-r--r-- 1 mhx users 18282075 Dec 10 21:17 perl-install-small-l9.dwarfs
+-rw-r--r-- 1 mhx users 35328512 Dec 8 14:25 perl-install-small.cromfs
+-rw-r--r-- 1 mhx users 25175016 Dec 10 21:14 perl-install-small.dwarfs
+```
It takes about 15 seconds longer to build the DwarFS file system with LZMA
compression (this is still 35 times faster than Cromfs), but reduces the
@@ -1203,84 +1330,94 @@ supports LZ4 compression.
I was feeling lucky and decided to run it on the full Perl dataset:
- $ time mkfs.erofs perl-install.erofs install -zlz4hc,9 -d2
- mkfs.erofs 1.2
- c_version: [ 1.2]
- c_dbg_lvl: [ 2]
- c_dry_run: [ 0]
- ^C
-
- real 912m42.601s
- user 903m2.777s
- sys 1m52.812s
+```
+$ time mkfs.erofs perl-install.erofs install -zlz4hc,9 -d2
+mkfs.erofs 1.2
+ c_version: [ 1.2]
+ c_dbg_lvl: [ 2]
+ c_dry_run: [ 0]
+^C
+
+real 912m42.601s
+user 903m2.777s
+sys 1m52.812s
+```
As you can tell, after more than 15 hours I just gave up. In those
15 hours, `mkfs.erofs` had produced a 13 GiB output file:
- $ ll -h perl-install.erofs
- -rw-r--r-- 1 mhx users 13G Dec 9 14:42 perl-install.erofs
+```
+$ ll -h perl-install.erofs
+-rw-r--r-- 1 mhx users 13G Dec 9 14:42 perl-install.erofs
+```
I don't think this would have been very useful to compare with DwarFS.
Just as for Cromfs, I re-ran with the smaller Perl dataset:
- $ time mkfs.erofs perl-install-small.erofs install-small -zlz4hc,9 -d2
- mkfs.erofs 1.2
- c_version: [ 1.2]
- c_dbg_lvl: [ 2]
- c_dry_run: [ 0]
-
- real 0m27.844s
- user 0m20.570s
- sys 0m1.848s
+```
+$ time mkfs.erofs perl-install-small.erofs install-small -zlz4hc,9 -d2
+mkfs.erofs 1.2
+ c_version: [ 1.2]
+ c_dbg_lvl: [ 2]
+ c_dry_run: [ 0]
+
+real 0m27.844s
+user 0m20.570s
+sys 0m1.848s
+```
That was surprisingly quick, which makes me think that, again, there
might be some accidentally quadratic complexity hiding in `mkfs.erofs`.
The output file it produced is an order of magnitude larger than the
DwarFS image:
- $ ls -l perl-install-small.*fs
- -rw-r--r-- 1 mhx users 26928161 Dec 8 15:05 perl-install-small.dwarfs
- -rw-r--r-- 1 mhx users 296488960 Dec 9 14:45 perl-install-small.erofs
+```
+$ ls -l perl-install-small.*fs
+-rw-r--r-- 1 mhx users 26928161 Dec 8 15:05 perl-install-small.dwarfs
+-rw-r--r-- 1 mhx users 296488960 Dec 9 14:45 perl-install-small.erofs
+```
Admittedly, this isn't a fair comparison. EROFS has a fixed block size
of 4 KiB, and it uses LZ4 compression. If we tweak DwarFS to the same
parameters, we get:
- $ time mkdwarfs -i install-small -o perl-install-small-lz4.dwarfs -C lz4hc:level=9 -S 12
- 21:21:18.136796 scanning install-small
- 21:21:18.376998 waiting for background scanners...
- 21:21:18.770703 assigning directory and link inodes...
- 21:21:18.776422 finding duplicate files...
- 21:21:18.903505 saved 267.8 MiB / 611.8 MiB in 22842/26401 duplicate files
- 21:21:18.903621 waiting for inode scanners...
- 21:21:19.676420 assigning device inodes...
- 21:21:19.677400 assigning pipe/socket inodes...
- 21:21:19.678014 building metadata...
- 21:21:19.678101 building blocks...
- 21:21:19.678116 saving names and links...
- 21:21:19.678306 ordering 3559 inodes using nilsimsa similarity...
- 21:21:19.678566 nilsimsa: depth=20000, limit=255
- 21:21:19.684227 pre-sorted index (3360 name, 2127 path lookups) [5.592ms]
- 21:21:19.685550 updating name and link indices...
- 21:21:19.810401 3559 inodes ordered [132ms]
- 21:21:19.810519 waiting for segmenting/blockifying to finish...
- 21:21:26.773913 saving chunks...
- 21:21:26.776832 saving directories...
- 21:21:26.821085 waiting for compression to finish...
- 21:21:27.020929 compressed 611.8 MiB to 140.7 MiB (ratio=0.230025)
- 21:21:27.036202 filesystem created without errors [8.899s]
- ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
- waiting for block compression to finish
- 3334 dirs, 0/0 soft/hard links, 26401/26401 files, 0 other
- original size: 611.8 MiB, dedupe: 267.8 MiB (22842 files), segment: 0 B
- filesystem: 344 MiB in 88073 blocks (91628 chunks, 3559/3559 inodes)
- compressed filesystem: 88073 blocks/140.7 MiB written
- ████████████████████████████████████████████████████████████████▏100% |
-
- real 0m9.075s
- user 0m37.718s
- sys 0m2.427s
+```
+$ time mkdwarfs -i install-small -o perl-install-small-lz4.dwarfs -C lz4hc:level=9 -S 12
+21:21:18.136796 scanning install-small
+21:21:18.376998 waiting for background scanners...
+21:21:18.770703 assigning directory and link inodes...
+21:21:18.776422 finding duplicate files...
+21:21:18.903505 saved 267.8 MiB / 611.8 MiB in 22842/26401 duplicate files
+21:21:18.903621 waiting for inode scanners...
+21:21:19.676420 assigning device inodes...
+21:21:19.677400 assigning pipe/socket inodes...
+21:21:19.678014 building metadata...
+21:21:19.678101 building blocks...
+21:21:19.678116 saving names and links...
+21:21:19.678306 ordering 3559 inodes using nilsimsa similarity...
+21:21:19.678566 nilsimsa: depth=20000, limit=255
+21:21:19.684227 pre-sorted index (3360 name, 2127 path lookups) [5.592ms]
+21:21:19.685550 updating name and link indices...
+21:21:19.810401 3559 inodes ordered [132ms]
+21:21:19.810519 waiting for segmenting/blockifying to finish...
+21:21:26.773913 saving chunks...
+21:21:26.776832 saving directories...
+21:21:26.821085 waiting for compression to finish...
+21:21:27.020929 compressed 611.8 MiB to 140.7 MiB (ratio=0.230025)
+21:21:27.036202 filesystem created without errors [8.899s]
+⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
+waiting for block compression to finish
+3334 dirs, 0/0 soft/hard links, 26401/26401 files, 0 other
+original size: 611.8 MiB, dedupe: 267.8 MiB (22842 files), segment: 0 B
+filesystem: 344 MiB in 88073 blocks (91628 chunks, 3559/3559 inodes)
+compressed filesystem: 88073 blocks/140.7 MiB written
+████████████████████████████████████████████████████████████████▏100% |
+
+real 0m9.075s
+user 0m37.718s
+sys 0m2.427s
+```
It finishes in less than half the time and produces an output image
that's half the size of the EROFS image.
diff --git a/doc/dwarfs-format.md b/doc/dwarfs-format.md
index 3c3ed4a5..9f511ab0 100644
--- a/doc/dwarfs-format.md
+++ b/doc/dwarfs-format.md
@@ -1,11 +1,9 @@
-dwarfs-format(5) -- DwarFS File System Format v2.3
-==================================================
+# dwarfs-format(5) -- DwarFS File System Format v2.3
## DESCRIPTION
This document describes the DwarFS file system format, version 2.3.
-
## FILE STRUCTURE
A DwarFS file system image is just a sequence of blocks. Each block has the
@@ -65,25 +63,23 @@ A couple of notes:
larger than the one it supports. However, a new program will still
read all file systems with a smaller minor version number.
-
### Section Types
There are currently 3 different section types.
- * `BLOCK` (0):
- A block of data. This is where all file data is stored. There can be
- an arbitrary number of blocks of this type.
+- `BLOCK` (0):
+ A block of data. This is where all file data is stored. There can be
+ an arbitrary number of blocks of this type.
- * `METADATA_V2_SCHEMA` (7):
- The schema used to layout the `METADATA_V2` block contents. This is
- stored in "compact" thrift encoding.
-
- * `METADATA_V2` (8):
- This section contains the bulk of the metadata. It's essentially just
- a collection of bit-packed arrays and structures. The exact layout of
- each list and structure depends on the actual data and is stored
- separately in `METADATA_V2_SCHEMA`.
+- `METADATA_V2_SCHEMA` (7):
+ The schema used to layout the `METADATA_V2` block contents. This is
+ stored in "compact" thrift encoding.
+- `METADATA_V2` (8):
+ This section contains the bulk of the metadata. It's essentially just
+ a collection of bit-packed arrays and structures. The exact layout of
+ each list and structure depends on the actual data and is stored
+ separately in `METADATA_V2_SCHEMA`.
## METADATA FORMAT
@@ -169,17 +165,12 @@ list. The index into this list is the `inode_num` from `dir_entries`,
but you can perform direct lookups based on the inode number as well.
The `inodes` list is strictly in the following order:
-* directory inodes (`S_IFDIR`)
-
-* symlink inodes (`S_IFLNK`)
-
-* regular *unique* file inodes (`S_IREG`)
-
-* regular *shared* file inodes (`S_IREG`)
-
-* character/block device inodes (`S_IFCHR`, `S_IFBLK`)
-
-* socket/pipe inodes (`S_IFSOCK`, `S_IFIFO`)
+- directory inodes (`S_IFDIR`)
+- symlink inodes (`S_IFLNK`)
+- regular *unique* file inodes (`S_IREG`)
+- regular *shared* file inodes (`S_IREG`)
+- character/block device inodes (`S_IFCHR`, `S_IFBLK`)
+- socket/pipe inodes (`S_IFSOCK`, `S_IFIFO`)
The offsets can thus be found by using a binary search with a
predicate on the inode more. The shared file offset can be found
@@ -287,7 +278,7 @@ is true.
The `directories` table, when stored in packed format, omits
all `parent_entry` fields and uses delta compression for the
-`first_entry` fields.
+`first_entry` fields.
In order to unpack all information, you first have to delta-
decompress the `first_entry` fields, then traverse the whole
diff --git a/doc/dwarfs.md b/doc/dwarfs.md
index a2fb4491..198902bb 100644
--- a/doc/dwarfs.md
+++ b/doc/dwarfs.md
@@ -1,5 +1,4 @@
-dwarfs(1) -- mount highly compressed read-only file system
-==========================================================
+# dwarfs(1) -- mount highly compressed read-only file system
## SYNOPSIS
@@ -14,103 +13,105 @@ but it has some distinct features.
Other than that, it's pretty straightforward to use. Once you've created a
file system image using mkdwarfs(1), you can mount it with:
- dwarfs image.dwarfs /path/to/mountpoint
+```
+dwarfs image.dwarfs /path/to/mountpoint
+```
## OPTIONS
In addition to the regular FUSE options, `dwarfs` supports the following
options:
- * `-o cachesize=`*value*:
- Size of the block cache, in bytes. You can append suffixes
- (`k`, `m`, `g`) to specify the size in KiB, MiB and GiB,
- respectively. Note that this is not the upper memory limit
- of the process, as there may be blocks in flight that are
- not stored in the cache. Also, each block that hasn't been
- fully decompressed yet will carry decompressor state along
- with it, which can use a significant amount of additional
- memory. For more details, see mkdwarfs(1).
+- `-o cachesize=`*value*:
+ Size of the block cache, in bytes. You can append suffixes
+ (`k`, `m`, `g`) to specify the size in KiB, MiB and GiB,
+ respectively. Note that this is not the upper memory limit
+ of the process, as there may be blocks in flight that are
+ not stored in the cache. Also, each block that hasn't been
+ fully decompressed yet will carry decompressor state along
+ with it, which can use a significant amount of additional
+ memory. For more details, see mkdwarfs(1).
- * `-o workers=`*value*:
- Number of worker threads to use for decompressing blocks.
- If you have a lot of CPUs, increasing this number can help
- speed up access to files in the filesystem.
+- `-o workers=`*value*:
+ Number of worker threads to use for decompressing blocks.
+ If you have a lot of CPUs, increasing this number can help
+ speed up access to files in the filesystem.
- * `-o decratio=`*value*:
- The ratio over which a block is fully decompressed. Blocks
- are only decompressed partially, so each block has to carry
- the decompressor state with it until it is fully decompressed.
- However, if a certain fraction of the block has already been
- decompressed, it may be beneficial to just decompress the rest
- and free the decompressor state. This value determines the
- ratio at which we fully decompress the block rather than
- keeping a partially decompressed block. A value of 0.8 means
- that as long as we've decompressed less than 80% of the block,
- we keep the partially decompressed block, but if we've
- decompressed more then 80%, we'll fully decompress it.
+- `-o decratio=`*value*:
+ The ratio over which a block is fully decompressed. Blocks
+ are only decompressed partially, so each block has to carry
+ the decompressor state with it until it is fully decompressed.
+ However, if a certain fraction of the block has already been
+ decompressed, it may be beneficial to just decompress the rest
+ and free the decompressor state. This value determines the
+ ratio at which we fully decompress the block rather than
+ keeping a partially decompressed block. A value of 0.8 means
+ that as long as we've decompressed less than 80% of the block,
+ we keep the partially decompressed block, but if we've
+ decompressed more then 80%, we'll fully decompress it.
- * `-o offset=`*value*|`auto`:
- Specify the byte offset at which the filesystem is located in
- the image, or use `auto` to detect the offset automatically.
- This is only useful for images that have some header located
- before the actual filesystem data.
+- `-o offset=`*value*|`auto`:
+ Specify the byte offset at which the filesystem is located in
+ the image, or use `auto` to detect the offset automatically.
+ This is only useful for images that have some header located
+ before the actual filesystem data.
- * `-o mlock=none`|`try`|`must`:
- Set this to `try` or `must` instead of the default `none` to
- try or require `mlock()`ing of the file system metadata into
- memory.
+- `-o mlock=none`|`try`|`must`:
+ Set this to `try` or `must` instead of the default `none` to
+ try or require `mlock()`ing of the file system metadata into
+ memory.
- * `-o enable_nlink`:
- Set this option if you want correct hardlink counts for regular
- files. If this is not specified, the hardlink count will be 1.
- Enabling this will slow down the initialization of the fuse
- driver as the hardlink counts will be determined by a full
- file system scan (it only takes about a millisecond to scan
- through 100,000 files, so this isn't dramatic). The fuse driver
- will also consume more memory to hold the hardlink count table.
- This will be 4 bytes for every regular file inode.
+- `-o enable_nlink`:
+ Set this option if you want correct hardlink counts for regular
+ files. If this is not specified, the hardlink count will be 1.
+ Enabling this will slow down the initialization of the fuse
+ driver as the hardlink counts will be determined by a full
+ file system scan (it only takes about a millisecond to scan
+ through 100,000 files, so this isn't dramatic). The fuse driver
+ will also consume more memory to hold the hardlink count table.
+ This will be 4 bytes for every regular file inode.
- * `-o readonly`:
- Show all file system entries as read-only. By default, DwarFS
- will preserve the original writeability, which is obviously a
- lie as it's a read-only file system. However, this is needed
- for overlays to work correctly, as otherwise directories are
- seen as read-only by the overlay and it'll be impossible to
- create new files even in a writeable overlay. If you don't use
- overlays and want the file system to reflect its read-only
- state, you can set this option.
+- `-o readonly`:
+ Show all file system entries as read-only. By default, DwarFS
+ will preserve the original writeability, which is obviously a
+ lie as it's a read-only file system. However, this is needed
+ for overlays to work correctly, as otherwise directories are
+ seen as read-only by the overlay and it'll be impossible to
+ create new files even in a writeable overlay. If you don't use
+ overlays and want the file system to reflect its read-only
+ state, you can set this option.
- * `-o (no_)cache_image`:
- By default, `dwarfs` tries to ensure that the compressed file
- system image will not be cached by the kernel (i.e. the default
- is `-o no_cache_image`). This will reduce the memory consumption
- of the FUSE driver to slightly more than the `cachesize`, plus
- the size of the metadata block. This usually isn't a problem,
- especially when the image is stored on an SSD, but if you want
- to maximize performance it can be beneficial to use
- `-o cache_image` to keep the compressed image data in the kernel
- cache.
+- `-o (no_)cache_image`:
+ By default, `dwarfs` tries to ensure that the compressed file
+ system image will not be cached by the kernel (i.e. the default
+ is `-o no_cache_image`). This will reduce the memory consumption
+ of the FUSE driver to slightly more than the `cachesize`, plus
+ the size of the metadata block. This usually isn't a problem,
+ especially when the image is stored on an SSD, but if you want
+ to maximize performance it can be beneficial to use
+ `-o cache_image` to keep the compressed image data in the kernel
+ cache.
- * `-o (no_)cache_files`:
- By default, files in the mounted file system will be cached by
- the kernel (i.e. the default is `-o cache_files`). This will
- significantly improve performance when accessing the same files
- over and over again, especially if the data from these files has
- been (partially) evicted from the block cache. By setting the
- `-o no_cache_files` option, you can force the fuse driver to not
- use the kernel cache for file data. If you're short on memory and
- only infrequently accessing files, this can be worth trying, even
- though it's likely that the kernel will already do the right thing
- even when the cache is enabled.
+- `-o (no_)cache_files`:
+ By default, files in the mounted file system will be cached by
+ the kernel (i.e. the default is `-o cache_files`). This will
+ significantly improve performance when accessing the same files
+ over and over again, especially if the data from these files has
+ been (partially) evicted from the block cache. By setting the
+ `-o no_cache_files` option, you can force the fuse driver to not
+ use the kernel cache for file data. If you're short on memory and
+ only infrequently accessing files, this can be worth trying, even
+ though it's likely that the kernel will already do the right thing
+ even when the cache is enabled.
- * `-o debuglevel=`*name*:
- Use this for different levels of verbosity along with either
- the `-f` or `-d` FUSE options. This can give you some insight
- over what the file system driver is doing internally, but it's
- mainly meant for debugging and the `debug` and `trace` levels
- in particular will slow down the driver.
+- `-o debuglevel=`*name*:
+ Use this for different levels of verbosity along with either
+ the `-f` or `-d` FUSE options. This can give you some insight
+ over what the file system driver is doing internally, but it's
+ mainly meant for debugging and the `debug` and `trace` levels
+ in particular will slow down the driver.
-* `-o tidy_strategy=`*name*:
+- `-o tidy_strategy=`*name*:
Use one of the following strategies to tidy the block cache:
- `none`:
@@ -128,14 +129,14 @@ options:
cache is traversed and all blocks that have been fully or
partially swapped out by the kernel will be removed.
-* `-o tidy_interval=`*time*:
+- `-o tidy_interval=`*time*:
Used only if `tidy_strategy` is not `none`. This is the interval
at which the cache tidying thread wakes up to look for blocks
that can be removed from the cache. This must be an integer value.
Suffixes `ms`, `s`, `m`, `h` are supported. If no suffix is given,
the value will be assumed to be in seconds.
-* `-o tidy_max_age=`*time*:
+- `-o tidy_max_age=`*time*:
Used only if `tidy_strategy` is `time`. A block will be removed
from the cache if it hasn't been used for this time span. This must
be an integer value. Suffixes `ms`, `s`, `m`, `h` are supported.
@@ -145,14 +146,14 @@ There's two particular FUSE options that you'll likely need at some
point, e.g. when trying to set up an `overlayfs` mount on top of
a DwarFS image:
- * `-o allow_root` and `-o allow_other`:
- These will ensure that the mounted file system can be read by
- either `root` or any other user in addition to the user that
- started the fuse driver. So if you're running `dwarfs` as a
- non-privileged user, you want to `-o allow_root` in case `root`
- needs access, for example when you're trying to use `overlayfs`
- along with `dwarfs`. If you're running `dwarfs` as `root`, you
- need `allow_other`.
+- `-o allow_root` and `-o allow_other`:
+ These will ensure that the mounted file system can be read by
+ either `root` or any other user in addition to the user that
+ started the fuse driver. So if you're running `dwarfs` as a
+ non-privileged user, you want to `-o allow_root` in case `root`
+ needs access, for example when you're trying to use `overlayfs`
+ along with `dwarfs`. If you're running `dwarfs` as `root`, you
+ need `allow_other`.
## TIPS & TRICKS
@@ -193,28 +194,34 @@ set of Perl versions back.
Here's what you need to do:
- * Create a set of directories. In my case, these are all located
- in `/tmp/perl` as this was the orginal install location.
+- Create a set of directories. In my case, these are all located
+ in `/tmp/perl` as this was the orginal install location.
- cd /tmp/perl
- mkdir install-ro
- mkdir install-rw
- mkdir install-work
- mkdir install
+ ```
+ cd /tmp/perl
+ mkdir install-ro
+ mkdir install-rw
+ mkdir install-work
+ mkdir install
+ ```
- * Mount the DwarFS image. `-o allow_root` is needed to make sure
- `overlayfs` has access to the mounted file system. In order
- to use `-o allow_root`, you may have to uncomment or add
- `user_allow_other` in `/etc/fuse.conf`.
+- Mount the DwarFS image. `-o allow_root` is needed to make sure
+ `overlayfs` has access to the mounted file system. In order
+ to use `-o allow_root`, you may have to uncomment or add
+ `user_allow_other` in `/etc/fuse.conf`.
- dwarfs perl-install.dwarfs install-ro -o allow_root
+ ```
+ dwarfs perl-install.dwarfs install-ro -o allow_root
+ ```
- * Now set up `overlayfs`.
+- Now set up `overlayfs`.
- sudo mount -t overlay overlay -o lowerdir=install-ro,upperdir=install-rw,workdir=install-work install
+ ```
+ sudo mount -t overlay overlay -o lowerdir=install-ro,upperdir=install-rw,workdir=install-work install
+ ```
- * That's it. You should now be able to access a writeable version
- of your DwarFS image in `install`.
+- That's it. You should now be able to access a writeable version
+ of your DwarFS image in `install`.
You can go even further than that. Say you have different sets of
modules that you regularly want to layer on top of the base DwarFS
@@ -223,7 +230,9 @@ the read-write directory after unmounting the `overlayfs`, and
selectively add this by passing a colon-separated list to the
`lowerdir` option when setting up the `overlayfs` mount:
- sudo mount -t overlay overlay -o lowerdir=install-ro:install-modules install
+```
+sudo mount -t overlay overlay -o lowerdir=install-ro:install-modules install
+```
If you want *this* merged overlay to be writable, just add in the
`upperdir` and `workdir` options from before again.
diff --git a/doc/dwarfsck.md b/doc/dwarfsck.md
index 092b4142..c783de75 100644
--- a/doc/dwarfsck.md
+++ b/doc/dwarfsck.md
@@ -1,5 +1,4 @@
-dwarfsck(1) -- check DwarFS image
-=================================
+# dwarfsck(1) -- check DwarFS image
## SYNOPSIS
@@ -15,43 +14,43 @@ with a non-zero exit code.
## OPTIONS
- * `-i`, `--input=`*file*:
- Path to the filesystem image.
+- `-i`, `--input=`*file*:
+ Path to the filesystem image.
- * `-d`, `--detail=`*value*:
- Level of filesystem information detail. The default is 2. Higher values
- mean more output. Values larger than 6 will currently not provide any
- further detail.
+- `-d`, `--detail=`*value*:
+ Level of filesystem information detail. The default is 2. Higher values
+ mean more output. Values larger than 6 will currently not provide any
+ further detail.
- * `-O`, `--image-offset=`*value*|`auto`:
- Specify the byte offset at which the filesystem is located in the image.
- Use `auto` to detect the offset automatically. This is also the default.
- This is only useful for images that have some header located before the
- actual filesystem data.
+- `-O`, `--image-offset=`*value*|`auto`:
+ Specify the byte offset at which the filesystem is located in the image.
+ Use `auto` to detect the offset automatically. This is also the default.
+ This is only useful for images that have some header located before the
+ actual filesystem data.
- * `-H`, `--print-header`:
- Print the header located before the filesystem image to stdout. If no
- header is present, the program will exit with a non-zero exit code.
+- `-H`, `--print-header`:
+ Print the header located before the filesystem image to stdout. If no
+ header is present, the program will exit with a non-zero exit code.
- * `-n`, `--num-workers=`*value*:
- Number of worker threads used for integrity checking.
+- `-n`, `--num-workers=`*value*:
+ Number of worker threads used for integrity checking.
- * `--check-integrity`:
- In addition to performing a fast checksum check, also perform a (much
- slower) verification of the embedded SHA-512/256 hashes.
+- `--check-integrity`:
+ In addition to performing a fast checksum check, also perform a (much
+ slower) verification of the embedded SHA-512/256 hashes.
- * `--json`:
- Print a simple JSON representation of the filesystem metadata. Please
- note that the format is *not* stable.
+- `--json`:
+ Print a simple JSON representation of the filesystem metadata. Please
+ note that the format is *not* stable.
- * `--export-metadata=`*file*:
- Export all filesystem meteadata in JSON format.
+- `--export-metadata=`*file*:
+ Export all filesystem meteadata in JSON format.
- * `--log-level=`*name*:
- Specifiy a logging level.
+- `--log-level=`*name*:
+ Specifiy a logging level.
- * `--help`:
- Show program help, including option defaults.
+- `--help`:
+ Show program help, including option defaults.
## AUTHOR
diff --git a/doc/dwarfsextract.md b/doc/dwarfsextract.md
index f7739483..d1fe458b 100644
--- a/doc/dwarfsextract.md
+++ b/doc/dwarfsextract.md
@@ -1,9 +1,8 @@
-dwarfsextract(1) -- extract DwarFS image
-========================================
+# dwarfsextract(1) -- extract DwarFS image
## SYNOPSIS
-`dwarfsextract` `-i` *image* [`-o` *dir*] [*options*...]
+`dwarfsextract` `-i` *image* [`-o` *dir*] [*options*...]
`dwarfsextract` `-i` *image* -f *format* [`-o` *file*] [*options*...]
## DESCRIPTION
@@ -35,44 +34,44 @@ to disk:
## OPTIONS
- * `-i`, `--input=`*file*:
- Path to the source filesystem.
+- `-i`, `--input=`*file*:
+ Path to the source filesystem.
- * `-o`, `--output=`*directory*|*file*:
- If no format is specified, this is the directory to which the contents
- of the filesystem should be extracted. If a format is specified, this
- is the name of the output archive. This option can be omitted, in which
- case the default is to extract the files to the current directory, or
- to write the archive data to stdout.
+- `-o`, `--output=`*directory*|*file*:
+ If no format is specified, this is the directory to which the contents
+ of the filesystem should be extracted. If a format is specified, this
+ is the name of the output archive. This option can be omitted, in which
+ case the default is to extract the files to the current directory, or
+ to write the archive data to stdout.
- * `-O`, `--image-offset=`*value*|`auto`:
- Specify the byte offset at which the filesystem is located in the image.
- Use `auto` to detect the offset automatically. This is also the default.
- This is only useful for images that have some header located before the
- actual filesystem data.
+- `-O`, `--image-offset=`*value*|`auto`:
+ Specify the byte offset at which the filesystem is located in the image.
+ Use `auto` to detect the offset automatically. This is also the default.
+ This is only useful for images that have some header located before the
+ actual filesystem data.
- * `-f`, `--format=`*format*:
- The archive format to produce. If this is left empty or unspecified,
- files will be extracted to the output directory (or the current directory
- if no output directory is specified). For a full list of supported formats,
- see libarchive-formats(5).
+- `-f`, `--format=`*format*:
+ The archive format to produce. If this is left empty or unspecified,
+ files will be extracted to the output directory (or the current directory
+ if no output directory is specified). For a full list of supported formats,
+ see libarchive-formats(5).
- * `-n`, `--num-workers=`*value*:
- Number of worker threads used for extracting the filesystem.
+- `-n`, `--num-workers=`*value*:
+ Number of worker threads used for extracting the filesystem.
- * `-s`, `--cache-size=`*value*:
- Size of the block cache, in bytes. You can append suffixes (`k`, `m`, `g`)
- to specify the size in KiB, MiB and GiB, respectively. Note that this is
- not the upper memory limit of the process, as there may be blocks in
- flight that are not stored in the cache. Also, each block that hasn't been
- fully decompressed yet will carry decompressor state along with it, which
- can use a significant amount of additional memory.
+- `-s`, `--cache-size=`*value*:
+ Size of the block cache, in bytes. You can append suffixes (`k`, `m`, `g`)
+ to specify the size in KiB, MiB and GiB, respectively. Note that this is
+ not the upper memory limit of the process, as there may be blocks in
+ flight that are not stored in the cache. Also, each block that hasn't been
+ fully decompressed yet will carry decompressor state along with it, which
+ can use a significant amount of additional memory.
- * `--log-level=`*name*:
- Specifiy a logging level.
+- `--log-level=`*name*:
+ Specifiy a logging level.
- * `--help`:
- Show program help, including option defaults.
+- `--help`:
+ Show program help, including option defaults.
## AUTHOR
diff --git a/doc/mkdwarfs.md b/doc/mkdwarfs.md
index 3c83c13f..a9a96504 100644
--- a/doc/mkdwarfs.md
+++ b/doc/mkdwarfs.md
@@ -1,9 +1,8 @@
-mkdwarfs(1) -- create highly compressed read-only file systems
-==============================================================
+# mkdwarfs(1) -- create highly compressed read-only file systems
## SYNOPSIS
-`mkdwarfs` `-i` *path* `-o` *file* [*options*...]
+`mkdwarfs` `-i` *path* `-o` *file* [*options*...]
`mkdwarfs` `-i` *file* `-o` *file* `--recompress` [*options*...]
## DESCRIPTION
@@ -26,272 +25,272 @@ After that, you can mount it with dwarfs(1):
There two mandatory options for specifying the input and output:
- * `-i`, `--input=`*path*|*file*:
- Path to the root directory containing the files from which you want to
- build a filesystem. If the `--recompress` option is given, this argument
- is the source filesystem.
+- `-i`, `--input=`*path*|*file*:
+ Path to the root directory containing the files from which you want to
+ build a filesystem. If the `--recompress` option is given, this argument
+ is the source filesystem.
- * `-o`, `--output=`*file*:
- File name of the output filesystem.
+- `-o`, `--output=`*file*:
+ File name of the output filesystem.
Most other options are concerned with compression tuning:
- * `-l`, `--compress-level=`*value*:
- Compression level to use for the filesystem. **If you are unsure, please
- stick to the default level of 7.** This is intended to provide some
- sensible defaults and will depend on which compression libraries were
- available at build time. **The default level has been chosen to provide
- you with the best possible compression while still keeping the file
- system very fast to access.** Levels 8 and 9 will switch to LZMA
- compression (when available), which will likely reduce the file system
- image size, but will make it about an order of magnitude slower to
- access, so reserve these levels for cases where you only need to access
- the data infrequently. This `-l` option is meant to be the "easy"
- interface to configure `mkdwarfs`, and it will actually pick defaults
- for seven distinct options: `--block-size-bits`, `--compression`,
- `--schema-compression`, `--metadata-compression`, `--window-size`,
- `--window-step` and `--order`. See the output of `mkdwarfs --help` for
- a table listing the exact defaults used for each compression level.
+- `-l`, `--compress-level=`*value*:
+ Compression level to use for the filesystem. **If you are unsure, please
+ stick to the default level of 7.** This is intended to provide some
+ sensible defaults and will depend on which compression libraries were
+ available at build time. **The default level has been chosen to provide
+ you with the best possible compression while still keeping the file
+ system very fast to access.** Levels 8 and 9 will switch to LZMA
+ compression (when available), which will likely reduce the file system
+ image size, but will make it about an order of magnitude slower to
+ access, so reserve these levels for cases where you only need to access
+ the data infrequently. This `-l` option is meant to be the "easy"
+ interface to configure `mkdwarfs`, and it will actually pick defaults
+ for seven distinct options: `--block-size-bits`, `--compression`,
+ `--schema-compression`, `--metadata-compression`, `--window-size`,
+ `--window-step` and `--order`. See the output of `mkdwarfs --help` for
+ a table listing the exact defaults used for each compression level.
- * `-S`, `--block-size-bits=`*value*:
- The block size used for the compressed filesystem. The actual block size
- is two to the power of this value. Larger block sizes will offer better
- overall compression ratios, but will be slower and consume more memory
- when actually using the filesystem, as blocks will have to be fully or at
- least partially decompressed into memory. Values between 20 and 26, i.e.
- between 1MiB and 64MiB, usually work quite well.
+- `-S`, `--block-size-bits=`*value*:
+ The block size used for the compressed filesystem. The actual block size
+ is two to the power of this value. Larger block sizes will offer better
+ overall compression ratios, but will be slower and consume more memory
+ when actually using the filesystem, as blocks will have to be fully or at
+ least partially decompressed into memory. Values between 20 and 26, i.e.
+ between 1MiB and 64MiB, usually work quite well.
- * `-N`, `--num-workers=`*value*:
- Number of worker threads used for building the filesystem. This defaults
- to the number of processors available on your system. Use this option if
- you want to limit the resources used by `mkdwarfs`.
- This option affects both the scanning phase and the compression phase.
- In the scanning phase, the worker threads are used to scan files in the
- background as they are discovered. File scanning includes checksumming
- for de-duplication as well as (optionally) checksumming for similarity
- computation, depending on the `--order` option. File discovery itself
- is single-threaded and runs independently from the scanning threads.
- In the compression phase, the worker threads are used to compress the
- individual filesystem blocks in the background. Ordering, segmenting
- and block building are, again, single-threaded and run independently.
+- `-N`, `--num-workers=`*value*:
+ Number of worker threads used for building the filesystem. This defaults
+ to the number of processors available on your system. Use this option if
+ you want to limit the resources used by `mkdwarfs`.
+ This option affects both the scanning phase and the compression phase.
+ In the scanning phase, the worker threads are used to scan files in the
+ background as they are discovered. File scanning includes checksumming
+ for de-duplication as well as (optionally) checksumming for similarity
+ computation, depending on the `--order` option. File discovery itself
+ is single-threaded and runs independently from the scanning threads.
+ In the compression phase, the worker threads are used to compress the
+ individual filesystem blocks in the background. Ordering, segmenting
+ and block building are, again, single-threaded and run independently.
- * `-B`, `--max-lookback-blocks=`*value*:
- Specify how many of the most recent blocks to scan for duplicate segments.
- By default, only the current block will be scanned. The larger this number,
- the more duplicate segments will likely be found, which may further improve
- compression. Impact on compression speed is minimal, but this could cause
- resulting filesystem to be slightly less efficient to use, as single small
- files can now potentially span multiple filesystem blocks. Passing `-B0`
- will completely disable duplicate segment search.
+- `-B`, `--max-lookback-blocks=`*value*:
+ Specify how many of the most recent blocks to scan for duplicate segments.
+ By default, only the current block will be scanned. The larger this number,
+ the more duplicate segments will likely be found, which may further improve
+ compression. Impact on compression speed is minimal, but this could cause
+ resulting filesystem to be slightly less efficient to use, as single small
+ files can now potentially span multiple filesystem blocks. Passing `-B0`
+ will completely disable duplicate segment search.
- * `-W`, `--window-size=`*value*:
- Window size of cyclic hash used for segmenting. This is again an exponent
- to a base of two. Cyclic hashes are used by `mkdwarfs` for finding
- identical segments across multiple files. This is done on top of duplicate
- file detection. If a reasonable amount of duplicate segments is found,
- this means less blocks will be used in the filesystem and potentially
- less memory will be used when accessing the filesystem. It doesn't
- necessarily mean that the filesystem will be much smaller, as this removes
- redundany that cannot be exploited by the block compression any longer.
- But it shouldn't make the resulting filesystem any bigger. This option
- is used along with `--window-step` to determine how extensive this
- segment search will be. The smaller the window sizes, the more segments
- will obviously be found. However, this also means files will become more
- fragmented and thus the filesystem can be slower to use and metadata
- size will grow. Passing `-W0` will completely disable duplicate segment
- search.
+- `-W`, `--window-size=`*value*:
+ Window size of cyclic hash used for segmenting. This is again an exponent
+ to a base of two. Cyclic hashes are used by `mkdwarfs` for finding
+ identical segments across multiple files. This is done on top of duplicate
+ file detection. If a reasonable amount of duplicate segments is found,
+ this means less blocks will be used in the filesystem and potentially
+ less memory will be used when accessing the filesystem. It doesn't
+ necessarily mean that the filesystem will be much smaller, as this removes
+ redundany that cannot be exploited by the block compression any longer.
+ But it shouldn't make the resulting filesystem any bigger. This option
+ is used along with `--window-step` to determine how extensive this
+ segment search will be. The smaller the window sizes, the more segments
+ will obviously be found. However, this also means files will become more
+ fragmented and thus the filesystem can be slower to use and metadata
+ size will grow. Passing `-W0` will completely disable duplicate segment
+ search.
- * `-w`, `--window-step=`*value*:
- This option specifies how often cyclic hash values are stored for lookup.
- It is specified relative to the window size, as a base-2 exponent that
- divides the window size. To give a concrete example, if `--window-size=16`
- and `--window-step=1`, then a cyclic hash across 65536 bytes will be stored
- at every 32768 bytes of input data. If `--window-step=2`, then a hash value
- will be stored at every 16384 bytes. This means that not every possible
- 65536-byte duplicate segment will be detected, but it is guaranteed that
- all duplicate segments of (`window_size` + `window_step`) bytes or more
- will be detected (unless they span across block boundaries, of course).
- If you use a larger value for this option, the increments become *smaller*,
- and `mkdwarfs` will be slightly slower and use more memory.
+- `-w`, `--window-step=`*value*:
+ This option specifies how often cyclic hash values are stored for lookup.
+ It is specified relative to the window size, as a base-2 exponent that
+ divides the window size. To give a concrete example, if `--window-size=16`
+ and `--window-step=1`, then a cyclic hash across 65536 bytes will be stored
+ at every 32768 bytes of input data. If `--window-step=2`, then a hash value
+ will be stored at every 16384 bytes. This means that not every possible
+ 65536-byte duplicate segment will be detected, but it is guaranteed that
+ all duplicate segments of (`window_size` + `window_step`) bytes or more
+ will be detected (unless they span across block boundaries, of course).
+ If you use a larger value for this option, the increments become *smaller*,
+ and `mkdwarfs` will be slightly slower and use more memory.
- * `--bloom-filter-size`=*value*:
- The segmenting algorithm uses a bloom filter to determine quickly if
- there is *no* match at a given position. This will filter out more than
- 90% of bad matches quickly with the default bloom filter size. The default
- is pretty much where the sweet spot lies. If you have copious amounts of
- RAM and CPU power, feel free to increase this by one or two and you *might*
- be able to see some improvement. If you're tight on memory, then decreasing
- this will potentially save a few MiBs.
+- `--bloom-filter-size`=*value*:
+ The segmenting algorithm uses a bloom filter to determine quickly if
+ there is *no* match at a given position. This will filter out more than
+ 90% of bad matches quickly with the default bloom filter size. The default
+ is pretty much where the sweet spot lies. If you have copious amounts of
+ RAM and CPU power, feel free to increase this by one or two and you *might*
+ be able to see some improvement. If you're tight on memory, then decreasing
+ this will potentially save a few MiBs.
- * `-L`, `--memory-limit=`*value*:
- Approximately how much memory you want `mkdwarfs` to use during filesystem
- creation. Note that currently this will only affect the block manager
- component, i.e. the number of filesystem blocks that are in flight but
- haven't been compressed and written to the output file yet. So the memory
- used by `mkdwarfs` can certainly be larger than this limit, but it's a
- good option when building large filesystems with expensive compression
- algorithms. Also note that most memory is likely used by the compression
- algorithms, so if you're short on memory it might be worth tweaking the
- compression options.
+- `-L`, `--memory-limit=`*value*:
+ Approximately how much memory you want `mkdwarfs` to use during filesystem
+ creation. Note that currently this will only affect the block manager
+ component, i.e. the number of filesystem blocks that are in flight but
+ haven't been compressed and written to the output file yet. So the memory
+ used by `mkdwarfs` can certainly be larger than this limit, but it's a
+ good option when building large filesystems with expensive compression
+ algorithms. Also note that most memory is likely used by the compression
+ algorithms, so if you're short on memory it might be worth tweaking the
+ compression options.
- * `-C`, `--compression=`*algorithm*[`:`*algopt*[`=`*value*][`,`...]]:
- The compression algorithm and configuration used for file system data.
- The value for this option is a colon-separated list. The first item is
- the compression algorithm, the remaining item are its options. Options
- can be either boolean or have a value. For details on which algori`thms
- and options are available, see the output of `mkdwarfs --help`. `zstd`
- will give you the best compression while still keeping decompression
- *very* fast. `lzma` will compress even better, but decompression will
- be around ten times slower.
+- `-C`, `--compression=`*algorithm*[`:`*algopt*[`=`*value*][`,`...]]:
+ The compression algorithm and configuration used for file system data.
+ The value for this option is a colon-separated list. The first item is
+ the compression algorithm, the remaining item are its options. Options
+ can be either boolean or have a value. For details on which algorithms
+ and options are available, see the output of `mkdwarfs --help`. `zstd`
+ will give you the best compression while still keeping decompression
+ *very* fast. `lzma` will compress even better, but decompression will
+ be around ten times slower.
- * `--schema-compression=`*algorithm*[`:`*algopt*[`=`*value*][`,`...]]:
- The compression algorithm and configuration used for the metadata schema.
- Takes the same arguments as `--compression` above. The schema is *very*
- small, in the hundreds of bytes, so this is only relevant for extremely
- small file systems. The default (`zstd`) has shown to give considerably
- better results than any other algorithms.
+- `--schema-compression=`*algorithm*[`:`*algopt*[`=`*value*][`,`...]]:
+ The compression algorithm and configuration used for the metadata schema.
+ Takes the same arguments as `--compression` above. The schema is *very*
+ small, in the hundreds of bytes, so this is only relevant for extremely
+ small file systems. The default (`zstd`) has shown to give considerably
+ better results than any other algorithms.
- * `--metadata-compression=`*algorithm*[`:`*algopt*[`=`*value*][`,`...]]:
- The compression algorithm and configuration used for the metadata.
- Takes the same arguments as `--compression` above. The metadata has been
- optimized for very little redundancy and leaving it uncompressed, the
- default for all levels below 7, has the benefit that it can be mapped
- to memory and used directly. This improves mount time for large file
- systems compared to e.g. an lzma compressed metadata block. If you don't
- care about mount time, you can safely choose `lzma` compression here, as
- the data will only have to be decompressed once when mounting the image.
+- `--metadata-compression=`*algorithm*[`:`*algopt*[`=`*value*][`,`...]]:
+ The compression algorithm and configuration used for the metadata.
+ Takes the same arguments as `--compression` above. The metadata has been
+ optimized for very little redundancy and leaving it uncompressed, the
+ default for all levels below 7, has the benefit that it can be mapped
+ to memory and used directly. This improves mount time for large file
+ systems compared to e.g. an lzma compressed metadata block. If you don't
+ care about mount time, you can safely choose `lzma` compression here, as
+ the data will only have to be decompressed once when mounting the image.
- * `--recompress`[`=all`|`=block`|`=metadata`|`=none`]:
- Take an existing DwarFS file system and recompress it using different
- compression algorithms. If no argument or `all` is given, all sections
- in the file system image will be recompressed. Note that *only* the
- compression algorithms, i.e. the `--compression`, `--schema-compression`
- and `--metadata-compression` options, have an impact on how the new file
- system is written. Other options, e.g. `--block-size-bits` or `--order`,
- have no impact. If `none` is given as an argument, none of the sections
- will be recompressed, but the file system is still rewritten in the
- latest file system format. This is an easy way of upgrading an old file
- system image to a new format. If `block` or `metadata` is given, only
- the block sections (i.e. the actual file data) or the metadata sections
- are recompressed. This can be useful if you want to switch from compressed
- metadata to uncompressed metadata without having to rebuild or recompress
- all the other data.
+- `--recompress`[`=all`|`=block`|`=metadata`|`=none`]:
+ Take an existing DwarFS file system and recompress it using different
+ compression algorithms. If no argument or `all` is given, all sections
+ in the file system image will be recompressed. Note that *only* the
+ compression algorithms, i.e. the `--compression`, `--schema-compression`
+ and `--metadata-compression` options, have an impact on how the new file
+ system is written. Other options, e.g. `--block-size-bits` or `--order`,
+ have no impact. If `none` is given as an argument, none of the sections
+ will be recompressed, but the file system is still rewritten in the
+ latest file system format. This is an easy way of upgrading an old file
+ system image to a new format. If `block` or `metadata` is given, only
+ the block sections (i.e. the actual file data) or the metadata sections
+ are recompressed. This can be useful if you want to switch from compressed
+ metadata to uncompressed metadata without having to rebuild or recompress
+ all the other data.
- * `-P`, `--pack-metadata=auto`|`none`|[`all`|`chunk_table`|`directories`|`shared_files`|`names`|`names_index`|`symlinks`|`symlinks_index`|`force`|`plain`[`,`...]]:
- Which metadata information to store in packed format. This is primarily
- useful when storing metadata uncompressed, as it allows for smaller
- metadata block size without having to turn on compression. Keep in mind,
- though, that *most* of the packed data must be unpacked into memory when
- reading the file system. If you want a purely memory-mappable metadata
- block, leave this at the default (`auto`), which will turn on `names` and
- `symlinks` packing if these actually help save data.
- Tweaking these options is mostly interesting when dealing with file
- systems that contain hundreds of thousands of files.
- See [Metadata Packing](#metadata-packing) for more details.
+- `-P`, `--pack-metadata=auto`|`none`|[`all`|`chunk_table`|`directories`|`shared_files`|`names`|`names_index`|`symlinks`|`symlinks_index`|`force`|`plain`[`,`...]]:
+ Which metadata information to store in packed format. This is primarily
+ useful when storing metadata uncompressed, as it allows for smaller
+ metadata block size without having to turn on compression. Keep in mind,
+ though, that *most* of the packed data must be unpacked into memory when
+ reading the file system. If you want a purely memory-mappable metadata
+ block, leave this at the default (`auto`), which will turn on `names` and
+ `symlinks` packing if these actually help save data.
+ Tweaking these options is mostly interesting when dealing with file
+ systems that contain hundreds of thousands of files.
+ See [Metadata Packing](#metadata-packing) for more details.
- * `--set-owner=`*uid*:
- Set the owner for all entities in the file system. This can reduce the
- size of the file system. If the input only has a single owner already,
- setting this won't make any difference.
+- `--set-owner=`*uid*:
+ Set the owner for all entities in the file system. This can reduce the
+ size of the file system. If the input only has a single owner already,
+ setting this won't make any difference.
- * `--set-group=`*gid*:
- Set the group for all entities in the file system. This can reduce the
- size of the file system. If the input only has a single group already,
- setting this won't make any difference.
+- `--set-group=`*gid*:
+ Set the group for all entities in the file system. This can reduce the
+ size of the file system. If the input only has a single group already,
+ setting this won't make any difference.
- * `--set-time=`*time*|`now`:
- Set the time stamps for all entities to this value. This can significantly
- reduce the size of the file system. You can pass either a unix time stamp
- or `now`.
+- `--set-time=`*time*|`now`:
+ Set the time stamps for all entities to this value. This can significantly
+ reduce the size of the file system. You can pass either a unix time stamp
+ or `now`.
- * `--keep-all-times`:
- As of release 0.3.0, by default, `mkdwarfs` will only save the contents of
- the `mtime` field in order to save metadata space. If you want to save
- `atime` and `ctime` as well, use this option.
+- `--keep-all-times`:
+ As of release 0.3.0, by default, `mkdwarfs` will only save the contents of
+ the `mtime` field in order to save metadata space. If you want to save
+ `atime` and `ctime` as well, use this option.
- * `--time-resolution=`*sec*|`sec`|`min`|`hour`|`day`:
- Specify the resolution with which time stamps are stored. By default,
- time stamps are stored with second resolution. You can specify "odd"
- resolutions as well, e.g. something like 15 second resolution is
- entirely possible. Moving from second to minute resolution, for example,
- will save roughly 6 bits per file system entry in the metadata block.
+- `--time-resolution=`*sec*|`sec`|`min`|`hour`|`day`:
+ Specify the resolution with which time stamps are stored. By default,
+ time stamps are stored with second resolution. You can specify "odd"
+ resolutions as well, e.g. something like 15 second resolution is
+ entirely possible. Moving from second to minute resolution, for example,
+ will save roughly 6 bits per file system entry in the metadata block.
- * `--order=none`|`path`|`similarity`|`nilsimsa`[`:`*limit*[`:`*depth*[`:`*mindepth*]]]|`script`:
- The order in which inodes will be written to the file system. Choosing `none`,
- the inodes will be stored in the order in which they are discovered. With
- `path`, they will be sorted asciibetically by path name of the first file
- representing this inode. With `similarity`, they will be ordered using a
- simple, yet fast and efficient, similarity hash function. `nilsimsa` ordering
- uses a more sophisticated similarity function that is typically better than
- `similarity`, but is significantly slower to compute. However, computation
- can happen in the background while already building the file system.
- `nilsimsa` ordering can be further tweaked by specifying a *limit* and
- *depth*. The *limit* determines how soon an inode is considered similar
- enough for adding. A *limit* of 255 means "essentially identical", whereas
- a *limit* of 0 means "not similar at all". The *depth* determines up to
- how many inodes can be checked at most while searching for a similar one.
- To avoid `nilsimsa` ordering to become a bottleneck when ordering lots of
- small files, the *depth* is adjusted dynamically to keep the input queue
- to the segmentation/compression stages adequately filled. You can specify
- how much the *depth* can be adjusted by also specifying *mindepth*.
- The default if you omit these values is a *limit* of 255, a *depth*
- of 20000 and a *mindepth* of 1000. Note that if you want reproducible
- results, you need to set *depth* and *mindepth* to the same value. Also
- note that when you're compressing lots (as in hundreds of thousands) of
- small files, ordering them by `similarity` instead of `nilsimsa` is likely
- going to speed things up significantly without impacting compression too much.
- Last but not least, if scripting support is built into `mkdwarfs`, you can
- choose `script` to let the script determine the order.
+- `--order=none`|`path`|`similarity`|`nilsimsa`[`:`*limit*[`:`*depth*[`:`*mindepth*]]]|`script`:
+ The order in which inodes will be written to the file system. Choosing `none`,
+ the inodes will be stored in the order in which they are discovered. With
+ `path`, they will be sorted asciibetically by path name of the first file
+ representing this inode. With `similarity`, they will be ordered using a
+ simple, yet fast and efficient, similarity hash function. `nilsimsa` ordering
+ uses a more sophisticated similarity function that is typically better than
+ `similarity`, but is significantly slower to compute. However, computation
+ can happen in the background while already building the file system.
+ `nilsimsa` ordering can be further tweaked by specifying a *limit* and
+ *depth*. The *limit* determines how soon an inode is considered similar
+ enough for adding. A *limit* of 255 means "essentially identical", whereas
+ a *limit* of 0 means "not similar at all". The *depth* determines up to
+ how many inodes can be checked at most while searching for a similar one.
+ To avoid `nilsimsa` ordering to become a bottleneck when ordering lots of
+ small files, the *depth* is adjusted dynamically to keep the input queue
+ to the segmentation/compression stages adequately filled. You can specify
+ how much the *depth* can be adjusted by also specifying *mindepth*.
+ The default if you omit these values is a *limit* of 255, a *depth*
+ of 20000 and a *mindepth* of 1000. Note that if you want reproducible
+ results, you need to set *depth* and *mindepth* to the same value. Also
+ note that when you're compressing lots (as in hundreds of thousands) of
+ small files, ordering them by `similarity` instead of `nilsimsa` is likely
+ going to speed things up significantly without impacting compression too much.
+ Last but not least, if scripting support is built into `mkdwarfs`, you can
+ choose `script` to let the script determine the order.
- * `--remove-empty-dirs`:
- Removes all empty directories from the output file system, recursively.
- This is particularly useful when using scripts that filter out a lot of
- file system entries.
+- `--remove-empty-dirs`:
+ Removes all empty directories from the output file system, recursively.
+ This is particularly useful when using scripts that filter out a lot of
+ file system entries.
- * `--with-devices`:
- Include character and block devices in the output file system. These are
- not included by default, and due to security measures in FUSE, they will
- never work in the mounted file system. However, they can still be copied
- out of the mounted file system, for example using `rsync`.
+- `--with-devices`:
+ Include character and block devices in the output file system. These are
+ not included by default, and due to security measures in FUSE, they will
+ never work in the mounted file system. However, they can still be copied
+ out of the mounted file system, for example using `rsync`.
- * `--with-specials`:
- Include named fifos and sockets in the output file system. These are not
- included by default.
+- `--with-specials`:
+ Include named fifos and sockets in the output file system. These are not
+ included by default.
- * `--header=`*file*:
- Read header from file and place it before the output filesystem image.
- Can be used with `--recompress` to add or replace a header.
+- `--header=`*file*:
+ Read header from file and place it before the output filesystem image.
+ Can be used with `--recompress` to add or replace a header.
- * `--remove-header`:
- Remove header from a filesystem image. Only useful with `--recompress`.
+- `--remove-header`:
+ Remove header from a filesystem image. Only useful with `--recompress`.
- * `--log-level=`*name*:
- Specifiy a logging level.
+- `--log-level=`*name*:
+ Specifiy a logging level.
- * `--no-progress`:
- Don't show progress output while building filesystem.
+- `--no-progress`:
+ Don't show progress output while building filesystem.
- * `--progress=none`|`simple`|`ascii`|`unicode`:
- Choosing `none` is equivalent to specifying `--no-progress`. `simple`
- will print a single line of progress information whenever the progress
- has significantly changed, but at most once every 2 seconds. This is
- also the default when the output is not a tty. `unicode` is the default
- behaviour, which shows a nice progress bar and lots of additional
- information. If your terminal cannot deal with unicode characters,
- you can switch to `ascii`, which is like `unicode`, but looks less
- fancy.
+- `--progress=none`|`simple`|`ascii`|`unicode`:
+ Choosing `none` is equivalent to specifying `--no-progress`. `simple`
+ will print a single line of progress information whenever the progress
+ has significantly changed, but at most once every 2 seconds. This is
+ also the default when the output is not a tty. `unicode` is the default
+ behaviour, which shows a nice progress bar and lots of additional
+ information. If your terminal cannot deal with unicode characters,
+ you can switch to `ascii`, which is like `unicode`, but looks less
+ fancy.
- * `--help`:
- Show program help, including defaults, compression level detail and
- supported compression algorithms.
+- `--help`:
+ Show program help, including defaults, compression level detail and
+ supported compression algorithms.
If experimental Python support was compiled into `mkdwarfs`, you can use the
following option to enable customizations via the scripting interface:
- * `--script=`*file*[`:`*class*[`(`arguments`...)`]]:
- Specify the Python script to load. The class name is optional if there's
- a class named `mkdwarfs` in the script. It is also possible to pass
- arguments to the constuctor.
+- `--script=`*file*[`:`*class*[`(`arguments`...)`]]:
+ Specify the Python script to load. The class name is optional if there's
+ a class named `mkdwarfs` in the script. It is also possible to pass
+ arguments to the constuctor.
## TIPS & TRICKS
@@ -342,70 +341,70 @@ However, there are several options to choose from that allow you to
further reduce metadata size without having to compress the metadata.
These options are controlled by the `--pack-metadata` option.
- * `auto`:
- This is the default. It will enable both `names` and `symlinks`.
+- `auto`:
+ This is the default. It will enable both `names` and `symlinks`.
- * `none`:
- Don't enable any packing. However, string tables (i.e. names and
- symlinks) will still be stored in "compact" rather than "plain"
- format. In order to force storage in plain format, use `plain`.
+- `none`:
+ Don't enable any packing. However, string tables (i.e. names and
+ symlinks) will still be stored in "compact" rather than "plain"
+ format. In order to force storage in plain format, use `plain`.
- * `all`:
- Enable all packing options. This does *not* force packing of
- string tables (i.e. names and symlinks) if the packing would
- actually increase the size, which can happen if the string tables
- are actually small. In order to force string table packing, use
- `all,force`.
+- `all`:
+ Enable all packing options. This does *not* force packing of
+ string tables (i.e. names and symlinks) if the packing would
+ actually increase the size, which can happen if the string tables
+ are actually small. In order to force string table packing, use
+ `all,force`.
- * `chunk_table`:
- Delta-compress chunk tables. This can reduce the size of the
- chunk tables for large file systems and help compression, however,
- it will likely require a lot of memory when unpacking the tables
- again. Only use this if you know what you're doing.
+- `chunk_table`:
+ Delta-compress chunk tables. This can reduce the size of the
+ chunk tables for large file systems and help compression, however,
+ it will likely require a lot of memory when unpacking the tables
+ again. Only use this if you know what you're doing.
- * `directories`:
- Pack directories table by storing first entry pointers delta-
- compressed and completely removing parent directory pointers.
- The parent directory pointers can be rebuilt by tree traversal
- when the filesystem is loaded. If you have a large number of
- directories, this can reduce the metadata size, however, it
- will likely require a lot of memory when unpacking the tables
- again. Only use this if you know what you're doing.
+- `directories`:
+ Pack directories table by storing first entry pointers delta-
+ compressed and completely removing parent directory pointers.
+ The parent directory pointers can be rebuilt by tree traversal
+ when the filesystem is loaded. If you have a large number of
+ directories, this can reduce the metadata size, however, it
+ will likely require a lot of memory when unpacking the tables
+ again. Only use this if you know what you're doing.
- * `shared_files`:
- Pack shared files table. This is only useful if the filesystem
- contains lots of non-hardlinked duplicates. It gets more efficient
- the more copies of a file are in the filesystem.
+- `shared_files`:
+ Pack shared files table. This is only useful if the filesystem
+ contains lots of non-hardlinked duplicates. It gets more efficient
+ the more copies of a file are in the filesystem.
- * `names`,`symlinks`:
- Compress the names and symlink targets using the
- [fsst](https://github.com/cwida/fsst) compression scheme. This
- compresses each individual entry separately using a small,
- custom symbol table, and it's surprisingly efficient. It is
- not uncommon for names to make up for 50-70% of the metadata,
- and fsst compression typically reduces the size by a factor
- of two. The entries can be decompressed individually, so no
- extra memory is used when accessing the filesystem (except for
- the symbol table, which is only a few hundred bytes). This is
- turned on by default. For small filesystems, it's possible that
- the compressed strings plus symbol table are actually larger
- than the uncompressed strings. If this is the case, the strings
- will be stored uncompressed, unless `force` is also specified.
+- `names`,`symlinks`:
+ Compress the names and symlink targets using the
+ [fsst](https://github.com/cwida/fsst) compression scheme. This
+ compresses each individual entry separately using a small,
+ custom symbol table, and it's surprisingly efficient. It is
+ not uncommon for names to make up for 50-70% of the metadata,
+ and fsst compression typically reduces the size by a factor
+ of two. The entries can be decompressed individually, so no
+ extra memory is used when accessing the filesystem (except for
+ the symbol table, which is only a few hundred bytes). This is
+ turned on by default. For small filesystems, it's possible that
+ the compressed strings plus symbol table are actually larger
+ than the uncompressed strings. If this is the case, the strings
+ will be stored uncompressed, unless `force` is also specified.
- * `names_index`,`symlinks_index`:
- Delta-compress the names and symlink targets indices. The same
- caveats apply as for `chunk_table`.
+- `names_index`,`symlinks_index`:
+ Delta-compress the names and symlink targets indices. The same
+ caveats apply as for `chunk_table`.
- * `force`:
- Forces the compression of the `names` and `symlinks` tables,
- even if that would make them use more memory than the
- uncompressed tables. This is really only useful for testing
- and development.
+- `force`:
+ Forces the compression of the `names` and `symlinks` tables,
+ even if that would make them use more memory than the
+ uncompressed tables. This is really only useful for testing
+ and development.
- * `plain`:
- Store string tables in "plain" format. The plain format uses
- Frozen thrift arrays and was used in earlier metadata versions.
- It is useful for debugging, but wastes up to one byte per string.
+- `plain`:
+ Store string tables in "plain" format. The plain format uses
+ Frozen thrift arrays and was used in earlier metadata versions.
+ It is useful for debugging, but wastes up to one byte per string.
To give you an idea of the metadata size using different packing options,
here's the size of the metadata block for the Ubuntu 20.04.2.0 Desktop
@@ -430,7 +429,6 @@ further compress the block. So if you're really desperately trying
to reduce the image size, enabling `all` packing would be an option
at the cost of using a lot more memory when using the filesystem.
-
## INTERNAL OPERATION
Internally, `mkdwarfs` runs in two completely separate phases. The first