Marcus Holland-Moritz
1b0d34accc
feat(logging): enable logging context at verbose level
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
912bd25a4a
feat(logging): more colours to distinguish log levels
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
92d6691b74
fix(block_merger): correctly compute queueable size
...
Don't assume that the block size doesn't change during a block's
lifetime. Blocks can be compressed in the background and shrink in
size. So instead of subtracting the block size from a queueable_size_
member and later adding a (potentially smaller) size back, we always
compute the queueable size from the queued size (by traversing all
queues), the releaseable size, and the maximum queueable size.
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
071da719cb
feat(filesystem_writer): use fsblock policy for block merger
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
c2ceeee9bd
docs(block_merger): add TODO about holder lifetime
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
6ca388fa6b
feat(block_merger): support block policies and partial release
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
a284aecc35
fix: update header size for Wave 64 format
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
cebfea15c1
chore: another attempt at fixing rebuilds
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
32da9c7577
chore: include run number/attempt when storing artifacts
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
8cf0642c3a
chore: pass some more action variable to CMakeLists.txt
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
fbad13553d
feat: allow setting worker group cpu affinity
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
85bc819fa7
refactor: add debug logging in multi_queue_block_merger
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
4ad155e19f
refactor: ostream/format support for fragment_category
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
d938191248
feat: integrate block merger for deterministic file system writing
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
cfb894e192
refactor: keep code for dumping file systems in case of errors
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
80b11347e1
fix: output threshold comparison in test logger
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
956294b6b9
refactor: check source validity in multi_queue_block_merger
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
69ffdf3e28
refactor: make merged_block_holder more flexible
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
181427c6ab
refactor: expose block_holder_type in multi_queue_block_merger
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
25d5390704
refactor: add safety checks to block manager
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
36cbd83ba4
refactor: replace block_merger is_last with separate call
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
442e7b8fd5
refactor: use callback for delayed updating of physical block numbers
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
a37c2716c8
feat: make number of segmenter worker threads configurable
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
18cd20ea03
fix: use correct worker pool, otherwise we can stall
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
5579b35a52
refactor: use holders to simplify releasing merged blocks
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
65a954b681
refactor: naming in multi_queue_block_merger
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
20e4a83b92
feat: add multi queue block merger + unit tests
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
f2249f3b6c
doc: update TODOs
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
d59ff62ad7
build: add example for setting per-file compile options
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
fb8c01cf18
Clean up console writer
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
f8e7f66b6a
Output metadata size in verbose mode
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
bff7e1b4b0
Support validation function in integral_value_parser
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
5046930d17
Use unsigned bytes in cyclic hash
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
3dd82bc085
Use zstd instead of lz4 for incompressible categorizer
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
b101694d06
Fix segmenter for the case when segmenting is disable
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
bad2c5e057
Factor out zstd context manager
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
9ac06fe40d
More verbose segmenter logging
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
f9a1db48ee
Simplify and speed up nilsimsa computation
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
b4becd5d48
Simplify multiversioning using target_clones()
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
f82fb8509e
Add nilsimsa update benchmark
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
acc4e7d7b4
Remove unused nilsimsa code
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
54e5723f01
Move progress contexts to bottom of output
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
5f3209028d
Merge chunks
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
79d8641998
Check for mergeable chunks and refactor chunk analysis
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
8eb4188277
Support operator[] for chunk_range
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
1ab991f8aa
Fix & refine hashing/scanning/categorizing progress
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
bfacd2e962
Suppress warnings with multiversioning
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
dd7d984ebf
Slightly increase nilsimsa parameters
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
166fa057c2
Cache bit vector pointers and factor out distance ordering
2023-12-17 23:02:28 +01:00
Marcus Holland-Moritz
743587e0cc
Tiny tweak to similarity ordering
2023-12-17 23:02:28 +01:00
