Further improve the way the near-optimal parser estimates symbol costs:
- When setting a block's initial costs, weigh the default costs and
previous block's costs differently, depending on how different the
current block seems to be from the previous block.
- When determining the "default" costs, take into account how many
literals appear in the block and how frequent matches seem to be.
- Increase BIT_COST from 8 to 16, to increase precision in calculations.
When the near-optimal parser sets the initial costs for a block, it
takes into account the costs of the previous block (if there is one).
However, the costs for the previous block were not being updated
following the final codes being built, so the costs from the last
optimization pass were being used instead of the final costs.
Make it so that the final costs are used, as intended.
Also, include the DEFLATE_END_OF_BLOCK symbol in the non-final codes.
In general, these changes improve the compression ratio slightly.
With deflate_compress_near_optimal(), some data benefits more than
originally thought from larger values of max_search_depth and
nice_match_length. Some data even needs these parameters to be fairly
high for deflate_compress_near_optimal() to compress more than
deflate_compress_lazy2(). Bump these parameters up a bit.
In the greedy and lazy compressors, automatically increase the minimum
match length from the default of 3 if the data doesn't contain many
different literals. This greatly improves the compression ratio of
levels 1-9 on certain types of data, such as DNA sequencing data, while
not worsening the ratio on other types of data.
The near-optimal compressor (used by compression levels 10-12) continues
to use a minimum match length of 3, since it already did a better job at
deciding when short matches are worthwhile. (The method for setting the
initial costs needs improvement; later commits address that.)
Resolves https://github.com/ebiggers/libdeflate/issues/57
Instead of switching directly from the lazy compressor at level 7 to the
near-optimal compressor at level 8, use the lazy2 compressor at levels
8-9 and don't switch to near-optimal until level 10.
This avoids poor compression ratio and bad performance (both
significantly worse than level 7, and significantly worse than zlib) at
levels 8-9 on data where the near-optimal compressor doesn't do well
until the parameters are cranked up.
On data where the near-optimal compressor *does* do well, this change
worsens the compression ratio of levels 8-9, but also speeds them up a
lot, thus positioning them similarly vs. zlib as the lower levels (i.e.
much faster and slightly stronger, rather than slightly faster and much
stronger). The difference between levels 9 and 10 is increased, but
that's perhaps the least bad place to have a discontinuity.
Resolves https://github.com/ebiggers/libdeflate/issues/85
Use a lower max_search_depth but a higher nice_match_length. This seems
to turn out a bit better, on average. This is consistent with what the
other compression levels do; level 4 was the only one that had
nice_match_length <= max_search_depth.
The new match scoring method in the lazy compressor has improved the
compression ratio slightly. Therefore, for levels 5-6 decrease
max_search_depth slightly to get a bit more performance.
We pull the version out of libdeflate.h into the
Makefile, and then sed a few macros out of the new
file libdeflate.pc.in. Set the necessary Cflags
and Libs (CFLAGS and LFLAGS) based off compile-time
definitions. Depend on the Makefile to pick up
version changes. Update uninstall target.
Signed-off-by: nick black <dankamongmen@gmail.com>
Environment variable DISABLE_SHARED (following convention of --disable-shared
of ./configure script) disables building of shared library and shared lib
symlink. It makes life of downstream maintainer easier when maintaining package
for environment that supports only static libraries.
See https://github.com/NixOS/nixpkgs/pull/144438
The '-t' option of GNU gzip allows checking whether a gzip file is valid
without writing the data anywhere. It's relatively straightforward to
support in libdeflate-gzip too, so add support for it.
Resolves https://github.com/ebiggers/libdeflate/issues/125
[EB - updated commit message]
A lot of the internal library headers don't have include guards because
they aren't needed. It might look like a bug, though, and it doesn't
hurt to add them. So do this.
Update https://github.com/ebiggers/libdeflate/issues/117
I saw this tweet claiming this flag makes libdeflate run 20% faster on
WebAssembly: https://twitter.com/Algunenano/status/1317098341377900550.
Indeed, when tried even in a complex PNG compression benchmark I've
observed 10-15% improvement when this flag is enabled.
Even though WebAssembly might be running on top of a variety of
underlying platforms, the spec requires it to support unaligned access,
and on majority of platforms it will translate to a faster code.
Hence, I think it makes sense to enable this flag by default.
Include sys/types.h to avoid the following build failure on uclibc:
In file included from programs/gzip.c:28:0:
programs/prog_util.h:159:1: error: unknown type name ‘ssize_t’
ssize_t xread(struct file_stream *strm, void *buf, size_t count);
^
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
