It doesn't seem worthwhile to have bt_matchfinder_get_matches() return
the best_len separately anymore, especially since it doesn't work as
expected due to it not handling length 3 matches.
Ensure that whenever deflate_choose_*() is called to choose a literal or
match, observe_*() is also called to tally it in the block split
statistics (except in deflate_compress_fastest(), which doesn't use the
block split statistics). Previously, the lazy compressor contained an
optimization that made the block split statistics differ from the actual
lazy parse. But that optimization no longer seems to be worthwhile.
For fastest, greedy, lazy, and lazy2: save memory by reducing the length
of the sequence store, and forcing a split if it is filled.
For fastest: increase the max block length, but use a relatively short
sequence store that will cause shorter blocks to be used often.
For all: allow the final block to exceed the soft maximum length if it
avoids having to create a block below the minimum length.
Introduce deflate_compress_fastest(), and use it for level 1. It uses
the ht_matchfinder instead of the hc_matchfinder, and it skips the block
splitting algorithm. This speeds up level 1, without changing the
compression ratio too much (relative to what is expected for level 1).
Further improve the way the near-optimal parser estimates symbol costs:
- When setting a block's initial costs, weigh the default costs and
previous block's costs differently, depending on how different the
current block seems to be from the previous block.
- When determining the "default" costs, take into account how many
literals appear in the block and how frequent matches seem to be.
- Increase BIT_COST from 8 to 16, to increase precision in calculations.
When the near-optimal parser sets the initial costs for a block, it
takes into account the costs of the previous block (if there is one).
However, the costs for the previous block were not being updated
following the final codes being built, so the costs from the last
optimization pass were being used instead of the final costs.
Make it so that the final costs are used, as intended.
Also, include the DEFLATE_END_OF_BLOCK symbol in the non-final codes.
In general, these changes improve the compression ratio slightly.
With deflate_compress_near_optimal(), some data benefits more than
originally thought from larger values of max_search_depth and
nice_match_length. Some data even needs these parameters to be fairly
high for deflate_compress_near_optimal() to compress more than
deflate_compress_lazy2(). Bump these parameters up a bit.
In the greedy and lazy compressors, automatically increase the minimum
match length from the default of 3 if the data doesn't contain many
different literals. This greatly improves the compression ratio of
levels 1-9 on certain types of data, such as DNA sequencing data, while
not worsening the ratio on other types of data.
The near-optimal compressor (used by compression levels 10-12) continues
to use a minimum match length of 3, since it already did a better job at
deciding when short matches are worthwhile. (The method for setting the
initial costs needs improvement; later commits address that.)
Resolves https://github.com/ebiggers/libdeflate/issues/57
Instead of switching directly from the lazy compressor at level 7 to the
near-optimal compressor at level 8, use the lazy2 compressor at levels
8-9 and don't switch to near-optimal until level 10.
This avoids poor compression ratio and bad performance (both
significantly worse than level 7, and significantly worse than zlib) at
levels 8-9 on data where the near-optimal compressor doesn't do well
until the parameters are cranked up.
On data where the near-optimal compressor *does* do well, this change
worsens the compression ratio of levels 8-9, but also speeds them up a
lot, thus positioning them similarly vs. zlib as the lower levels (i.e.
much faster and slightly stronger, rather than slightly faster and much
stronger). The difference between levels 9 and 10 is increased, but
that's perhaps the least bad place to have a discontinuity.
Resolves https://github.com/ebiggers/libdeflate/issues/85
Use a lower max_search_depth but a higher nice_match_length. This seems
to turn out a bit better, on average. This is consistent with what the
other compression levels do; level 4 was the only one that had
nice_match_length <= max_search_depth.
The new match scoring method in the lazy compressor has improved the
compression ratio slightly. Therefore, for levels 5-6 decrease
max_search_depth slightly to get a bit more performance.
We pull the version out of libdeflate.h into the
Makefile, and then sed a few macros out of the new
file libdeflate.pc.in. Set the necessary Cflags
and Libs (CFLAGS and LFLAGS) based off compile-time
definitions. Depend on the Makefile to pick up
version changes. Update uninstall target.
Signed-off-by: nick black <dankamongmen@gmail.com>
Environment variable DISABLE_SHARED (following convention of --disable-shared
of ./configure script) disables building of shared library and shared lib
symlink. It makes life of downstream maintainer easier when maintaining package
for environment that supports only static libraries.
See https://github.com/NixOS/nixpkgs/pull/144438
