Skip to content

Commit

Permalink
Merge pull request #7 from dajhorn/libdeflate-v1.19
Browse files Browse the repository at this point in the history 
Update libdeflate v1.19
  • Loading branch information
@amadvance
amadvance authored Oct 10, 2023
2 parents 96ae466 + 269b6ef commit 9a1f013
Show file tree
Hide file tree
Showing 36 changed files with 4,446 additions and 897 deletions.
14 changes: 14 additions & 0 deletions Makefile.am
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ bin_PROGRAMS = advzip advpng advmng advdef
7z/WindowOut.cc

libdeflate_SOURCES = \
libdeflate/arm/cpu_features.c \
libdeflate/x86/cpu_features.c \
libdeflate/adler32.c \
libdeflate/crc32.c \
libdeflate/deflate_compress.c\
Expand Down Expand Up @@ -209,6 +211,18 @@ noinst_HEADERS = \
7z/RangeCoder.h \
7z/WindowIn.h \
7z/WindowOut.h \
libdeflate/arm/adler32_impl.h \
libdeflate/arm/cpu_features.h \
libdeflate/arm/crc32_impl.h \
libdeflate/arm/crc32_pmull_helpers.h \
libdeflate/arm/crc32_pmull_wide.h \
libdeflate/arm/matchfinder_impl.h \
libdeflate/x86/adler32_impl.h \
libdeflate/x86/cpu_features.h \
libdeflate/x86/crc32_impl.h \
libdeflate/x86/crc32_pclmul_template.h \
libdeflate/x86/decompress_impl.h \
libdeflate/x86/matchfinder_impl.h \
libdeflate/adler32_vec_template.h \
libdeflate/bt_matchfinder.h \
libdeflate/common_defs.h \
Expand Down
81 changes: 81 additions & 0 deletions libdeflate/NEWS.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,86 @@
# libdeflate release notes

## Version 1.19

* Added new functions `libdeflate_alloc_compressor_ex()` and
`libdeflate_alloc_decompressor_ex()`. These functions allow specifying a
custom memory allocator on a per-compressor basis.

* libdeflate now always generates Huffman codes with at least 2 codewords. This
fixes a compatibility issue where Windows Explorer's ZIP unpacker could not
decompress DEFLATE streams created by libdeflate. libdeflate's behavior was
allowed by the DEFLATE RFC, but not all software was okay with it. In rare
cases, compression ratios can be slightly reduced by this change.

* Disabled the use of some compiler intrinsics on MSVC versions where they don't
work correctly.

* libdeflate can now compress up to the exact size of the output buffer.

* Slightly improved compression performance at levels 1-9.

* Improved the compression ratio of very short inputs.

## Version 1.18

* Fixed a bug where the build type didn't default to "Release" when using
CMake 3.10 or earlier.

* Fixed a bug where some optimized code wasn't used when building with
Clang 15 or later (x86), or with Clang 16 or later (aarch64).

* Fixed build errors with some architecture and compiler combos:
* aarch64 with Clang 16
* armv6kz or armv7e-m with gcc
* armhf with gcc (on Debian only)

## Version 1.17

(Apologies for another release so soon after v1.16, but the bug fix listed below
needed to go out.)

* Fixed a bug introduced in v1.16 where compression at levels 10-12 would
sometimes produce an output larger than the size that was returned by the
corresponding `libdeflate_*_compress_bound()` function.

* Converted the fuzzing scripts to use LLVM's libFuzzer and added them to the
GitHub Actions workflow. (This would have detected the above bug.)

* Further improved the support for direct compilation without using the official
build system. The top-level source directory no longer needs to be added to
the include path, and building the programs no longer requires that
`_FILE_OFFSET_BITS` and `_POSIX_C_SOURCE` be defined on the command line.

## Version 1.16

* Improved the compression ratio at levels 10-12 slightly, mainly levels 11-12.
Some inputs (such as certain PNG files) see much improved compression ratios.
As a trade-off, compressing at levels 11-12 is now about 5-20% slower.

* For consistency with zlib, the decompressor now returns an error on some
invalid inputs that were accepted before.

* Fixed a build error on arm64 with gcc with certain target CPUs. (Fixes v1.12)

* Fixed a build error on arm32 with gcc 10.1-10.3 and 11.1-11.2. (Fixes v1.15)

* Fixed a build error on arm32 with gcc in soft float mode. (Fixes v1.15)

* Fixed a build error in programs/gzip.c with uClibc. (Fixes v1.15)

* Fixed the install target on Windows. (Fixes v1.15)

## Version 1.15

* libdeflate now uses CMake instead of a plain Makefile.

* Improved MSVC support. Enabled most architecture-specific code with MSVC,
fixed building with clang in MSVC compatibility mode, and other improvements.

* When libdeflate is built with MinGW, the static library and import library are
now named using the MinGW convention (`*.a` and `*.dll.a`) instead of the
Visual Studio convention. This affects the official Windows binaries.

## Version 1.14

Significantly improved decompression performance on all platforms. Examples
Expand Down
183 changes: 43 additions & 140 deletions libdeflate/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,178 +14,73 @@ library, both for compression and decompression, and especially on x86
processors. In addition, libdeflate provides optional high compression modes
that provide a better compression ratio than the zlib's "level 9".

libdeflate itself is a library, but the following command-line programs which
use this library are also provided:
libdeflate itself is a library. The following command-line programs which use
this library are also included:

* gzip (or gunzip), a program which mostly behaves like the standard equivalent,
except that it does not yet have good streaming support and therefore does not
yet support very large files
* benchmark, a program for benchmarking in-memory compression and decompression
* `libdeflate-gzip`, a program which can be a drop-in replacement for standard
`gzip` under some circumstances. Note that `libdeflate-gzip` has some
limitations; it is provided for convenience and is **not** meant to be the
main use case of libdeflate. It needs a lot of memory to process large files,
and it omits support for some infrequently-used options of GNU gzip.

* `benchmark`, a test program that does round-trip compression and decompression
of the provided data, and measures the compression and decompression speed.
It can use libdeflate, zlib, or a combination of the two.

* `checksum`, a test program that checksums the provided data with Adler-32 or
CRC-32, and optionally measures the speed. It can use libdeflate or zlib.

For the release notes, see the [NEWS file](NEWS.md).

## Table of Contents

- [Building](#building)
- [Using the Makefile](#using-the-makefile)
- [For UNIX](#for-unix)
- [For macOS](#for-macos)
- [For Windows](#for-windows)
- [Using Cygwin](#using-cygwin)
- [Using MSYS2](#using-msys2)
- [Using a custom build system](#using-a-custom-build-system)
- [Using CMake](#using-cmake)
- [Directly integrating the library sources](#directly-integrating-the-library-sources)
- [API](#api)
- [Bindings for other programming languages](#bindings-for-other-programming-languages)
- [DEFLATE vs. zlib vs. gzip](#deflate-vs-zlib-vs-gzip)
- [Compression levels](#compression-levels)
- [Motivation](#motivation)
- [License](#license)


# Building

libdeflate and the provided programs like `gzip` can be built using the provided
Makefile. If only the library is needed, it can alternatively be easily
integrated into applications and built using any build system; see [Using a
custom build system](#using-a-custom-build-system).

## Using the Makefile

### For UNIX
## Using CMake

Just run `make`, then (if desired) `make install`. You need GNU Make and either
GCC or Clang. GCC is recommended because it builds slightly faster binaries.
libdeflate uses [CMake](https://cmake.org/). It can be built just like any
other CMake project, e.g. with:

By default, the following targets are built: the static library `libdeflate.a`,
the shared library `libdeflate.so`, the `gzip` program, and the `gunzip` program
(which is actually just a hard link to `gzip`). Benchmarking and test programs
such as `benchmark` are not built by default. You can run `make help` to
display the available build targets.
cmake -B build && cmake --build build

There are also many options which can be set on the `make` command line, e.g. to
omit library features or to customize the directories into which `make install`
installs files. See the Makefile for details.
By default the following targets are built:

### For macOS
- The static library (normally called `libdeflate.a`)
- The shared library (normally called `libdeflate.so`)
- The `libdeflate-gzip` program, including its alias `libdeflate-gunzip`

Prebuilt macOS binaries can be installed with [Homebrew](https://brew.sh):

brew install libdeflate

But if you need to build the binaries yourself, see the section for UNIX above.

### For Windows
Besides the standard CMake build and installation options, there are some
libdeflate-specific build options. See `CMakeLists.txt` for the list of these
options. To set an option, add `-DOPTION=VALUE` to the `cmake` command.

Prebuilt Windows binaries can be downloaded from
https://github.com/ebiggers/libdeflate/releases. But if you need to build the
binaries yourself, MinGW (gcc) is the recommended compiler to use. If you're
performing the build *on* Windows (as opposed to cross-compiling for Windows on
Linux, for example), you'll need to follow the directions in **one** of the two
sections below to set up a minimal UNIX-compatible environment using either
Cygwin or MSYS2, then do the build. (Other MinGW distributions may not work, as
they often omit basic UNIX tools such as `sh`.)

Alternatively, libdeflate may be built using the Visual Studio toolchain by
running `nmake /f Makefile.msc`. However, while this is supported in the sense
that it will produce working binaries, it is not recommended because the
binaries built with MinGW will be significantly faster.

Also note that 64-bit binaries are faster than 32-bit binaries and should be
preferred whenever possible.

#### Using Cygwin

Run the Cygwin installer, available from https://cygwin.com/setup-x86_64.exe.
When you get to the package selection screen, choose the following additional
packages from category "Devel":

- git
- make
- mingw64-i686-binutils
- mingw64-i686-gcc-g++
- mingw64-x86_64-binutils
- mingw64-x86_64-gcc-g++

(You may skip the mingw64-i686 packages if you don't need to build 32-bit
binaries.)

After the installation finishes, open a Cygwin terminal. Then download
libdeflate's source code (if you haven't already) and `cd` into its directory:

git clone https://github.com/ebiggers/libdeflate
cd libdeflate

(Note that it's not required to use `git`; an alternative is to extract a .zip
or .tar.gz archive of the source code downloaded from the releases page.
Also, in case you need to find it in the file browser, note that your home
directory in Cygwin is usually located at `C:\cygwin64\home\<your username>`.)

Then, to build 64-bit binaries:
https://github.com/ebiggers/libdeflate/releases.

make CC=x86_64-w64-mingw32-gcc
## Directly integrating the library sources

or to build 32-bit binaries:

make CC=i686-w64-mingw32-gcc

#### Using MSYS2

Run the MSYS2 installer, available from http://www.msys2.org/. After
installing, open an MSYS2 shell and run:

pacman -Syu

Say `y`, then when it's finished, close the shell window and open a new one.
Then run the same command again:

pacman -Syu

Then, install the packages needed to build libdeflate:

pacman -S git \
make \
mingw-w64-i686-binutils \
mingw-w64-i686-gcc \
mingw-w64-x86_64-binutils \
mingw-w64-x86_64-gcc

(You may skip the mingw-w64-i686 packages if you don't need to build 32-bit
binaries.)

Then download libdeflate's source code (if you haven't already):

git clone https://github.com/ebiggers/libdeflate

(Note that it's not required to use `git`; an alternative is to extract a .zip
or .tar.gz archive of the source code downloaded from the releases page.
Also, in case you need to find it in the file browser, note that your home
directory in MSYS2 is usually located at `C:\msys64\home\<your username>`.)

Then, to build 64-bit binaries, open "MSYS2 MinGW 64-bit" from the Start menu
and run the following commands:

cd libdeflate
make clean
make

Or to build 32-bit binaries, do the same but use "MSYS2 MinGW 32-bit" instead.

## Using a custom build system

The source files of the library are designed to be compilable directly, without
any prerequisite step like running a `./configure` script. Therefore, as an
alternative to building the library using the provided Makefile, the library
source files can be easily integrated directly into your application and built
using any build system.
Although the official build system is CMake, care has been taken to keep the
library source files compilable directly, without a prerequisite configuration
step. Therefore, it is also fine to just add the library source files directly
to your application, without using CMake.

You should compile both `lib/*.c` and `lib/*/*.c`. You don't need to worry
about excluding irrelevant architecture-specific code, as this is already
handled in the source files themselves using `#ifdef`s.

It is **strongly** recommended to use either gcc or clang, and to use `-O2`.
It is strongly recommended to use either gcc or clang, and to use `-O2`.

If you are doing a freestanding build with `-ffreestanding`, you must add
`-DFREESTANDING` as well, otherwise performance will suffer greatly.
`-DFREESTANDING` as well (matching what the `CMakeLists.txt` does).

# API

Expand Down Expand Up @@ -221,6 +116,9 @@ following bindings:
* Go: [go-libdeflate](https://github.com/4kills/go-libdeflate)
* Java: [libdeflate-java](https://github.com/astei/libdeflate-java)
* Julia: [LibDeflate.jl](https://github.com/jakobnissen/LibDeflate.jl)
* Nim: [libdeflate-nim](https://github.com/gemesa/libdeflate-nim)
* Perl: [Gzip::Libdeflate](https://github.com/benkasminbullock/gzip-libdeflate)
* PHP: [ext-libdeflate](https://github.com/pmmp/ext-libdeflate)
* Python: [deflate](https://github.com/dcwatson/deflate)
* Ruby: [libdeflate-ruby](https://github.com/kaorimatz/libdeflate-ruby)
* Rust: [libdeflater](https://github.com/adamkewley/libdeflater)
Expand All @@ -229,6 +127,11 @@ Note: these are third-party projects which haven't necessarily been vetted by
the authors of libdeflate. Please direct all questions, bugs, and improvements
for these bindings to their authors.

Also, unfortunately many of these bindings bundle or pin an old version of
libdeflate. To avoid known issues in old versions and to improve performance,
before using any of these bindings please ensure that the bundled or pinned
version of libdeflate has been upgraded to the latest release.

# DEFLATE vs. zlib vs. gzip

The DEFLATE format ([rfc1951](https://www.ietf.org/rfc/rfc1951.txt)), the zlib
Expand Down
11 changes: 5 additions & 6 deletions libdeflate/adler32.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@
*/

#include "lib_common.h"
#include "libdeflate.h"

/* The Adler-32 divisor, or "base", value */
#define DIVISOR 65521
Expand Down Expand Up @@ -91,10 +90,10 @@ adler32_generic(u32 adler, const u8 *p, size_t len)
#undef DEFAULT_IMPL
#undef arch_select_adler32_func
typedef u32 (*adler32_func_t)(u32 adler, const u8 *p, size_t len);
#if defined(__arm__) || defined(__aarch64__)
//# include "arm/adler32_impl.h"
#elif defined(__i386__) || defined(__x86_64__)
//# include "x86/adler32_impl.h"
#if defined(ARCH_ARM32) || defined(ARCH_ARM64)
# include "arm/adler32_impl.h"
#elif defined(ARCH_X86_32) || defined(ARCH_X86_64)
# include "x86/adler32_impl.h"
#endif

#ifndef DEFAULT_IMPL
Expand Down Expand Up @@ -122,7 +121,7 @@ static u32 dispatch_adler32(u32 adler, const u8 *p, size_t len)
#define adler32_impl DEFAULT_IMPL
#endif

LIBDEFLATEEXPORT u32 LIBDEFLATEAPI
LIBDEFLATEAPI u32
libdeflate_adler32(u32 adler, const void *buffer, size_t len)
{
if (buffer == NULL) /* Return initial value. */
Expand Down
Loading

0 comments on commit 9a1f013

Please sign in to comment.