TheCloud/dwarfs
1
0
Fork 0
mirror of https://github.com/mhx/dwarfs.git synced 2025-09-14 06:48:39 -04:00
Code Issues Packages Projects Releases Wiki Activity

Update documentation

Browse Source
This commit is contained in:
Marcus Holland-Moritz 2021-03-28 00:23:00 +01:00
parent b7ec25b887
commit b4a8e0134b
1 changed files with 20 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
doc/mkdwarfs.md
View File

@ -48,24 +48,32 @@ Most other options are concerned with compression tuning:
access, so reserve these levels for cases where you only need to access access, so reserve these levels for cases where you only need to access
the data infrequently. This `-l` option is meant to be the "easy" the data infrequently. This `-l` option is meant to be the "easy"
interface to configure `mkdwarfs`, and it will actually pick defaults interface to configure `mkdwarfs`, and it will actually pick defaults
for six distinct options: `--block-size-bits`, `--compression`, for seven distinct options: `--block-size-bits`, `--compression`,
`--schema-compression`, `--metadata-compression`, `--window-size` and `--schema-compression`, `--metadata-compression`, `--window-size`,
`--order`. See the output of `mkdwarfs --help` for a table listing the `--window-step` and `--order`. See the output of `mkdwarfs --help` for
exact defaults used for each compression level. a table listing the exact defaults used for each compression level.
* `-S`, `--block-size-bits=`*value*: * `-S`, `--block-size-bits=`*value*:
The block size used for the compressed filesystem. The actual block size The block size used for the compressed filesystem. The actual block size
is two to the power of this value. The valid range of this option is from is two to the power of this value. Larger block sizes will offer better
12 to 28, i.e. block sizes between 4kiB and 256MiB. Larger block sizes overall compression ratios, but will be slower and consume more memory
will offer better compression, but will be slower and consume more memory
when actually using the filesystem, as blocks will have to be fully or at when actually using the filesystem, as blocks will have to be fully or at
least partially decompressed into memory. Value between 20 and 24, i.e. least partially decompressed into memory. Values between 20 and 26, i.e.
between 1MiB and 16MiB, are usually a good compromise. between 1MiB and 64MiB, usually work quite well.
* `-N`, `--num-workers=`*value*: * `-N`, `--num-workers=`*value*:
Number of worker threads used for building the filesystem. This defaults Number of worker threads used for building the filesystem. This defaults
to the number of processors available on your system. Use this option if to the number of processors available on your system. Use this option if
you want to limit the resources used by `mkdwarfs`. you want to limit the resources used by `mkdwarfs`.
This option affects both the scanning phase and the compression phase.
In the scanning phase, the worker threads are used to scan files in the
background as they are discovered. File scanning includes checksumming
for de-duplication as well as (optionally) checksumming for similarity
computation, depending on the `--order` option. File discovery itself
is single-threaded and runs independently from the scanning threads.
In the compression phase, the worker threads are used to compress the
individual filesystem blocks in the background. Ordering, segmenting
and block building are, again, single-threaded and run independently.
* `-L`, `--memory-limit=`*value*: * `-L`, `--memory-limit=`*value*:
Approximately how much memory you want `mkdwarfs` to use during filesystem Approximately how much memory you want `mkdwarfs` to use during filesystem
@ -74,7 +82,9 @@ Most other options are concerned with compression tuning:
haven't been compressed and written to the output file yet. So the memory haven't been compressed and written to the output file yet. So the memory
used by `mkdwarfs` can certainly be larger than this limit, but it's a used by `mkdwarfs` can certainly be larger than this limit, but it's a
good option when building large filesystems with expensive compression good option when building large filesystems with expensive compression
algorithms. algorithms. Also note that most memory is likely used by the compression
algorithms, so if you're short on memory it might be worth tweaking the
compression options.
* `-C`, `--compression=`*algorithm*[`:`*algopt*[`=`*value*][`,`...]]: * `-C`, `--compression=`*algorithm*[`:`*algopt*[`=`*value*][`,`...]]:
The compression algorithm and configuration used for file system data. The compression algorithm and configuration used for file system data.