Skip to content

Commit

Permalink
Rename DEFLATE64 -> Deflate64
Browse files Browse the repository at this point in the history
  • Loading branch information
brianhelba committed Feb 26, 2021
1 parent ba78bce commit 59abf96
Show file tree
Hide file tree
Showing 4 changed files with 17 additions and 17 deletions.
22 changes: 11 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# zipfile-deflate64
[![PyPI](https://img.shields.io/pypi/v/zipfile-deflate64)](https://pypi.org/project/zipfile-deflate64/)

Extract DEFLATE64 ZIP archives with Python's `zipfile` API.
Extract Deflate64 ZIP archives with Python's `zipfile` API.

## Installation
```bash
Expand All @@ -14,7 +14,7 @@ with [manylinux2014](https://github.com/pypa/manylinux), macOS and Windows wheel
## Usage
Anywhere in a Python codebase:
```python
import zipfile_deflate64 # This has the side effect of patching the zipfile module to support DEFLATE64
import zipfile_deflate64 # This has the side effect of patching the zipfile module to support Deflate64
```

Alternatively, `zipfile_deflate64` re-exports the `zipfile` API, as a convenience:
Expand All @@ -28,42 +28,42 @@ zipfile.ZipFile(...)
## Design Rationale
### The Problem
Recent versions of Microsoft Windows Explorer
[use DEFLATE64 compression when creating ZIP files larger than 2GB](https://github.com/dotnet/runtime/issues/17802#issuecomment-231808916).
[use Deflate64 compression when creating ZIP files larger than 2GB](https://github.com/dotnet/runtime/issues/17802#issuecomment-231808916).
With the ubiquity of Windows and the ease of using "Sent to compressed folder", a majority of newly-created large
ZIP files use DEFLATE64 compression.
ZIP files use Deflate64 compression.

However, **support for DEFLATE64 in the open-source ecosystem is awful**!
Most ZIP libraries have declined to implement DEFLATE64,
However, **support for Deflate64 in the open-source ecosystem is very poor**!
Most ZIP libraries have declined to implement Deflate64,
citing [its proprietary nature](https://en.wikipedia.org/wiki/Deflate#Deflate64/Enhanced_Deflate).

In the .NET ecosystem, the [`ZipArchive` API supports decompression only](https://github.com/dotnet/corefx/pull/11264).
In Java, the [Apache Commons Compress APIs support both compression and decompression](https://commons.apache.org/proper/commons-compress/examples.html#Archivers_and_Compressors).

The 7-Zip project probably provides the best general-purpose support for compressing and decompressing
DEFLATE64, but there are several obstacles to general usability:
Deflate64, but there are several obstacles to general usability:
* [7-Zip itself](https://www.7-zip.org/) is a Windows-only GUI application
* 7-Zip is still issuing new releases, but has declined to implement certain new compression formats,
so the [mcmilk/7-Zip-zstd](https://github.com/mcmilk/7-Zip-zstd) fork is notable.
* [p7zip, the POSIX-compatible CLI version](http://p7zip.sourceforge.net/) (which does include DEFLATE64),
* [p7zip, the POSIX-compatible CLI version](http://p7zip.sourceforge.net/) (which does include Deflate64),
[has not had a release since 2016 and is likely unmaintained](https://github.com/jinfeihan57/p7zip/issues/114#issuecomment-761551564).
* p7zip does not build an API for external software to invoke for decompression.
* p7zip seems to now be living on as the [jinfeihan57/p7zip](https://github.com/jinfeihan57/p7zip) fork,
which is packaged by Arch Linux, amongst others.
* This seems to be active, and now can be built with CMake, but there's no support for building an external API.
* Many re-implementations of 7-Zip, such as [py7zr](https://github.com/miurahr/py7zr) for Python, do not support
DEFLATE64.
Deflate64.

In the Python ecosystem in particular, there have been several unfulfilled requests (
[[1]](https://github.com/UCL-ShippingGroup/pyrate/issues/33)
[[2]](https://www.reddit.com/r/learnpython/comments/iqr6eb/zip_files_with_compression_type_9_deflate64/)
[[3]](https://stackoverflow.com/a/12809847)
) for DEFLATE64 decompression support.
) for Deflate64 decompression support.

### A Solution
The best hope seems to be the [infback9](https://github.com/madler/zlib/tree/master/contrib/infback9) extension
to zlib. This was developed by Mark Adler, the original author of zlib, and is kept in the source repository of zlib,
but it is not officially supported and contains no build tooling and is not distributed with zlib packages.
Additionally, infback9 provides only low-level support for working with DEFLATE64 bitstreams, with no support for
Additionally, infback9 provides only low-level support for working with Deflate64 bitstreams, with no support for
the ZIP archive format (which is out of scope for zlib).

infback9's C-language API is relatively simple, but requires a non-trivial struct and function pointers for
Expand Down
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

setup(
name='zipfile-deflate64',
description="Extract DEFLATE64 ZIP archives with Python's zipfile API.",
description="Extract Deflate64 ZIP archives with Python's zipfile API.",
long_description=long_description,
long_description_content_type='text/markdown',
license='Apache 2.0',
Expand Down
2 changes: 1 addition & 1 deletion tests/test_deflate64.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,5 +43,5 @@ def test_decompress_empty(deflate64):


def test_decompress_invalid(deflate64):
with pytest.raises(ValueError, match=r'^Bad DEFLATE64 data: '):
with pytest.raises(ValueError, match=r'^Bad Deflate64 data: '):
deflate64.decompress(b'garbage')
8 changes: 4 additions & 4 deletions zipfile_deflate64/deflate64/deflate64module.c
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@ static PyObject* Deflate64_decompress(Deflate64Object* self, PyObject *args) {
break;
case Z_DATA_ERROR:
// Deflate format error
PyErr_Format(PyExc_ValueError, "Bad DEFLATE64 data: %s", self->strm->msg);
PyErr_Format(PyExc_ValueError, "Bad Deflate64 data: %s", self->strm->msg);
goto error;
case Z_MEM_ERROR:
// Could not allocate memory for the state
Expand Down Expand Up @@ -206,14 +206,14 @@ static PyMemberDef Deflate64_members[] = {
};

static PyMethodDef Deflate64_methods[] = {
{"decompress", (PyCFunction) Deflate64_decompress, METH_VARARGS, "Decompress a DEFLATE64 stream."},
{"decompress", (PyCFunction) Deflate64_decompress, METH_VARARGS, "Decompress a Deflate64 stream."},
{NULL}
};

static PyTypeObject Deflate64_type = {
PyVarObject_HEAD_INIT(NULL, 0)
.tp_name = "zipfile_deflate64.deflate64.Deflate64",
.tp_doc = "An object for DEFLATE64 decompression.",
.tp_doc = "An object for Deflate64 decompression.",
.tp_basicsize = sizeof(Deflate64Object),
.tp_flags = Py_TPFLAGS_DEFAULT,
.tp_new = PyType_GenericNew,
Expand All @@ -226,7 +226,7 @@ static PyTypeObject Deflate64_type = {
static PyModuleDef deflate64_module = {
PyModuleDef_HEAD_INIT,
.m_name = "deflate64",
.m_doc = "Python access to zlib's infback9 extension for DEFLATE64 decompression.",
.m_doc = "Python access to zlib's infback9 extension for Deflate64 decompression.",
.m_size = -1,
};

Expand Down

0 comments on commit 59abf96

Please sign in to comment.