From 1b21be88042c6cebabefa61513049fc4a4f4964b Mon Sep 17 00:00:00 2001 From: Marcus Holland-Moritz Date: Wed, 12 Jul 2023 12:25:52 +0200 Subject: [PATCH] Add another performance example to README --- README.md | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/README.md b/README.md index 1cfdebf2..046ec0e1 100644 --- a/README.md +++ b/README.md @@ -53,6 +53,45 @@ than SquashFS compression**, it's **6 times faster to build the file system**, it's typically faster to access files on DwarFS and it uses less CPU resources. +To give you an idea of what DwarFS is capable of, here's a quick comparison +of DwarFS and SquashFS on a set of video files with a total size of 39 GiB. +The twist is that each unique video file has two sibling files with a +different set of audio streams (I didn't make this up, this is +[an actual use case](https://github.com/mhx/dwarfs/discussions/63)). So +there's redundancy in both the video and audio data, but as the streams +are interleaved and identical blocks are typically very far apart, it's +quite challenging to make use of that redundancy for compression. SquashFS +essentially fails to compress the source data at all, whereas DwarFS is +able to reduce the size by almost a factor of 3, which is close to the +theoretical maximum: + +``` +$ du -hs dwarfs-video-test +39G dwarfs-video-test +$ ls -lh dwarfs-video-test.*fs +-rw-r--r-- 1 mhx users 14G Jul 2 13:01 dwarfs-video-test.dwarfs +-rw-r--r-- 1 mhx users 39G Jul 12 09:41 dwarfs-video-test.squashfs +``` + +While this is already impressive, it gets even better. When mounting +the SquashFS image and performing a random-read throughput test using +[fio](https://github.com/axboe/fio/)-3.34, both `squashfuse` and +`squashfuse_ll` top out at around 230 MiB/s: + +``` +$ fio --readonly --rw=randread --name=randread --bs=64k --direct=1 \ + --opendir=mnt --numjobs=4 --ioengine=libaio --iodepth=32 \ + --group_reporting --runtime=60 --time_based +[...] + READ: bw=230MiB/s (241MB/s), 230MiB/s-230MiB/s (241MB/s-241MB/s), io=13.5GiB (14.5GB), run=60004-60004msec +``` + +DwarFS, however, manages to sustain **random read rates of 20 GiB/s**: + +``` + READ: bw=20.2GiB/s (21.7GB/s), 20.2GiB/s-20.2GiB/s (21.7GB/s-21.7GB/s), io=1212GiB (1301GB), run=60001-60001msec +``` + Distinct features of DwarFS are: - Clustering of files by similarity using a similarity hash function.