nattviken

hz — an archiver that knows what it is packing

github.com/hugowikstrom/hz

På svenska →

tar | zstd is two programs that know nothing about each other. tar emits files in whatever order the directory tree happens to hold them; the compressor sees the stream through a window and is never told that file 4000 is a copy of file 12, or that the nine megabytes of "noise" it just gave up on are really a PNG stream that can be unfolded.

hz knows. It classifies the content, stores duplicates once, sorts similar files next to each other, unfolds PNG deflate layers and finds the recipe that stitches them back bit-exactly, recodes JPEG Huffman layers to arithmetic coding without touching a pixel — and lets every block choose its codec by probing instead of assuming. Everything is lossless and proven at pack time: whatever cannot be reproduced exactly is stored raw.

How it works

Five things happen before a single byte is compressed, and a sixth while it is.

1. The content is classified. Every file is read once and given a class from its own content, not its file name: text, binary, PNG, JPEG or opaque — already compressed. The class decides everything that follows.

2. Identical files are stored once. Content is hashed, and duplicates become pointers into the same block. tar stores every copy and hopes the compressor finds them, which it only does if they happen to sit within the window. In the PNG corpus, 28.7 % of the bytes are duplicates.

3. Similar files are placed next to each other. Entries are sorted by class and extension, so all the JavaScript sits in one run and all the screenshots in another. How much that is worth depends entirely on the window and on how scattered the likeness was to begin with — on the corpora here it measures zero, and on a tree with its likeness thrown around, −8 %. See the sorting question below.

4. PNGs are unfolded and recompressed. A PNG is already deflate-compressed, and 1996-vintage deflate is a poor shell around image data: a 32 kB window, one file at a time. hz unfolds the IDAT stream to raw filter rows and compresses them together with every other image's raw data. The way back is the hard part — deflate is not canonical, the same input gives different output depending on parameters — so hz finds the recipe: the zlib parameter set that reproduces exactly the stream the file already has.

5. JPEGs are recoded without a pixel touched. A JPEG is DCT coefficients Huffman-coded, and Huffman cannot put a symbol in less than one whole bit. hz decodes the stream and recodes the same coefficients with an adaptive arithmetic coder that can. −15 % on real photos, and the image is the same down to the last bit.

6. Nothing is assumed — everything is probed. Every block chooses its codec by compressing a sample with each candidate: zstd, xz or the in-house context mixer nvcm. Every transform is also tried in both directions at pack time. Whatever cannot be reproduced exactly is stored raw. That is why "lossless" here is not a promise but a check that runs.

The command line is gzip's, because gzip's surface is the one your fingers already know: hz dir gives dir.nv, hz -d dir.nv gives the tree back, and -c -d -k -l -t -v -1…-9 mean what they mean in gzip. One difference is deliberate: a directory operand becomes one archive of the whole tree — that is what hz is for — and the source tree is always left in place.

Measured

corpusvs today's tar+zstd −3vs the best classical tool
log files−55.2 %−15.4 % vs xz −9e
images (GIF)−47.6 %±0
images (PNG)−42.9 %−15.5 %
enwik8−39.6 %−13.7 %
archives of archives−37.9 %−2.4 % vs zstd −22, −0.05 % vs LZMA2 at maximum
source code−37.1 %−12.5 %
Silesia−31.0 %−5.7 %
photos (JPEG)−15.1 %−14.8 % (no classical tool takes anything at all)
film−0.3 %±0 — but in a sixth of the time

And since 2026-11-18, smallest is not enough: on six of the nine corpora hz is both smaller and faster than the best established tool on the same machine. The report showing it is below.

The in-house compressor nvcm (context mixing, eight models voting per bit) beats xz -9e on both standard corpora: 1.677 bits/byte on Silesia and 1.713 on enwik8 — between the dictionary compressors and the heavy context mixers on the LTCB scale, in a 30 kB binary.

The full evaluation — eleven corpora, reconstruction proofs down to nanosecond timestamps, five kinds of deliberate damage — is in the repo in English and Swedish.

Report: hz against the established tools

18 November 2026.

The table above compares size and time separately. That is the easy question. The hard one is whether hz beats the same opponent on both axes at once: a smaller archive and a shorter pack time than the best established tool on that very corpus, measured on the same machine.

The opponent is therefore not chosen by us. For each corpus it is whichever of gzip, bzip2, zstd and xz produced the smallest archive, with its own measured pack time. Of hz's modes, the smallest of those that dominate is counted.

corpusbest establishedhzoutcome
source codexz -9e 8 629 516 B / 47 snormal 8 580 323 B / 40 s−0.57 % and 15 % faster
images (PNG)zstd -22 257 132 369 B / 156 snormal 230 452 176 B / 127 s−10.4 % and 18 % faster
photos (JPEG)bzip2 -9 424 611 032 B / 57 snormal 361 834 581 B / 48 s−14.8 % and 16 % faster
archives of archiveszstd -22 254 567 277 B / 154 snormal 254 287 214 B / 153 s−0.11 % and 1 % faster
log filesxz -9e 12 419 456 B / 897 ssmall 10 512 452 B / 191 s−15.4 % and 79 % faster
Silesiaxz -9e 48 408 228 B / 158 ssmall 45 657 473 B / 100 s−5.7 % and 37 % faster
images (GIF)zstd -22 14 867 185 B / 5 ssmall 14 864 575 B / 6 s−0.02 %, but 20 % more time
filmzstd -19 557 861 782 B / 57 snormal 557 880 433 B / 56 sfaster, but 18 kB larger
enwik8xz -9e 24 831 492 B / 121 ssmall 21 417 335 B / 129 s−13.8 %, but 7 % more time

Six of nine, and the three remaining are the three with nothing to win. GIF and film are already entropy-coded — 0.02 and 0.00 percent available — so only hz's fixed cost shows. enwik8 is a single file, which is worse than it sounds: hz cannot even split the work, because a block is filled by whole objects, so a 95 MB file always lands in one block. Deduplication, sorting and transforms have nothing to contribute, and what remains is the context mixer's price in time.

The boundary does not run between data types. It runs between exploitable structure and none.

Four measurements that moved something

The dictionary must cover the block. LZMA's preset 9e has a fixed 64 MiB dictionary, and hz's blocks are larger — so the compressor by construction could not see a repeat further back than 64 MiB, and that is precisely the repeat the deduplication and the sort put there. Measured on 120 MiB of archive data: 98 700 988 bytes with the 64 MiB dictionary, 96 559 784 with one covering the input. 2.17 % smaller and 12 % faster — a found match is cheaper than a continued search.

The sample has a blind spot. The codec probe takes 4 MiB as eight disjoint pieces of the block — the right construction for seeing what the data is, but it tears apart the long-range structure that separates the codecs. On a 120 MiB block from the archive corpus the probe chose zstd and got 100.13 MiB, where xz with a covering dictionary gave 91.97. The probe missed eight percent.

The register carried 16 hash bytes per file. The hash exists so that hz -t can detect mis-assembled content, and 32 bits suffice — bit errors are already caught by the blocks' CRC64. But hashes are random and therefore incompressible: on the code corpus they were 94 368 bytes of a 239 049-byte register. Eleven per mille of the whole archive, in a field nobody reads.

The cheap step was not cheap. The two-step rule for opaque blocks used zstd -19 to answer "does this block have tens of percent to give?". That question needs no strong compressor. zstd -3 --long=27 gives the same answer in 725 milliseconds instead of 48 seconds.

Three of the changes fixed regressions they had caused

This is the part worth carrying forward. The covering dictionary tied up 2.7 GB per encoder, and since the memory gate then admitted one at a time, the PNG corpus went from 268 to 841 seconds — for an archive 0.06 % smaller. A dictionary win that removes the parallelism is no win. The fix was embarrassingly simple in hindsight: a dictionary larger than the block cannot be used by construction, but it costs memory anyway. The floor was removed and the time became 127 s.

The same pattern twice more. Unconditionally re-testing the codec choice on the whole block doubled enwik8 from 131 to 259 seconds without saving a single byte — on text the context mixer wins the probe by tens of percent and wins the whole block too, so the re-check now runs only when the probe's lead was thin. And zstd -3 without the long window saw only 2.6 % of the archive corpus's 33: the escalation never fired and the archive grew from 248 to 271 MB, until --long=27 gave 27.6 % in the same 725 ms. The window was free; the level was expensive.

And a bug that was not in hz

In the middle of the re-measurement, two rows reported three files not coming back identical. The archive was flawless. It was the corpus that moved: the corpus builder hard-links instead of copying, so the file in korpus/kod is the same inode as the file in the live working tree — 5 971 of 5 974 files in the code corpus. Three files in a project tree were saved from another window while the measurement ran. The measurement harness now fingerprints the corpus before and after each run and refuses to interpret a run during which it moved, and the corpus can be frozen with --kopiera. But the consequence is broader than two rows: every number measured before that day rests on a corpus that could move.

What could not be won

Three things stand, and they belong in a report as much as the wins. The GPU mode still wins no block — LZ77 matching is 2.5× faster on the graphics card, but entropy coding is serial and Amdahl wins. Below 30 kB of input hz is larger than xz -9e, and the whole loss is the register: the payload is already 80 bytes smaller than xz's entire file. And the lossless JPEG recoding sits five percentage points below packJPG and Lepton, which have done the same thing since 2006 and 2016 — which is also the answer to everyone who asked whether any of this is new. No single method is. The packaging and the measurement are.

Frequently asked questions

These are not invented. They were asked when hz was posted, in the ninety-odd comments the post received. Most are objections, and some of them changed the code. The numbers can be tested: everything is in the report above and in the repo.

Can it be unpacked again? Yes, and it is the one requirement in the project treated as a requirement rather than a goal. Every transform is tried in both directions at pack time: whatever cannot be reproduced exactly is stored raw. The evidence is md5 manifests before and after on twelve thousand files, the Linux kernel with 86 618 of 86 618 files OK, a thousand randomized trees, and five kinds of deliberate damage that were all refused rather than returned as wrong data.

Is it the same method as zip? No, and partly yes. zip is deflate from 1993: LZ77 with a 32 kB window plus Huffman, one file at a time. hz has three layers where zip has one. Before compression comes what zip never does — duplicates stored once, similar files adjacent, PNG and JPEG compression layers removed so the raw data can be packed with everything else. The compression itself is usually not home-grown: it is libzstd and liblzma, the same libraries zstd and xz use, chosen per block by probing. In-house coders exist for two cases.

Does it use Huffman? The opposite — hz's own coders remove Huffman. A JPEG is DCT coefficients Huffman-coded, and Huffman cannot code a symbol in less than one whole bit. nvjpg decodes the stream and recodes the same coefficients with an adaptive binary arithmetic coder. That is where the 15 percent comes from, and it is also why no pixel is touched.

Does this break the entropy limit? No, and hz does not claim to. The limit is a property of the data under a model, not of the data. Everything in compression history since Shannon is model improvement, not a violation of the theorem. Concretely: on enwik8, where the literature has measured for thirty years, hz loses to paq8px — 20.4 against 15.8 MB. A tool that broke the limit would not trail a published context mixer by 29 %.

Don't you have to count the program itself? Yes — it is the right question to ask anyone claiming fantastic compression. The binary is ~430 kB, contains zero built-in dictionaries or trained tables, and does not grow with the corpus — nvcm builds its tables from the data at run time and throws them away. For comparison, xz plus liblzma is 316 kB. And the win exceeds the program: on the log corpus hz saves 1.8 MiB against xz −9e, several times the whole binary. The final proof is the kernel test regardless: a third-party corpus the program had never seen.

Isn't the win just sorting the files? No — and the measurement says so in a way that does not favour hz: with the compressor held constant, the sort is worth zero on both the code and the PNG corpus, slightly negative with a small window. It pays only when likeness is scattered and the window small — the same code files thrown into 200 random directories give −8.0 % with a 2 MiB window and −2.2 % with a 64 MiB dictionary. The method is also long known (gwern.net/sort). hz's measured wins come from the PNG/JPEG transforms, from deduplication where duplicates exist, and from the context mixer.

Is the result tuned to the test data? That objection cannot be answered with more of the same data, so the answer is data that is not ours: Linux kernel 6.12 fetched from kernel.org, and zlib-ng's corpora with Calgary and Canterbury — the literature's old standard corpora, which also consist of small files, hz's weakest area. Seven wins of eight there, and on the kernel −12.0 % against the best classical tool — the same order as on our own code corpus (−11.6 %). Plus a thousand randomized trees through the full round trip.

How large is the codebook? Zero bytes. hz has none. nvcm's model is adaptive — built from the data during packing and rebuilt identically during unpacking, so it never needs storing. The price is memory instead: about 537 MiB for nvcm, against xz −9e's 674 MiB.

What about small files, say 100 kB? The break-even against xz −9e sits around 30 kB. Below it hz loses (+14 % at 1 kB), above it wins (−12 % at 100 kB, −15 % at 1 MiB). The cause is not the compression but the register: at 1 kB hz's payload is already 80 bytes smaller than xz's whole file, but the register adds 175 bytes against xz's ~60.

Doesn't it take much longer? On six of nine corpora hz is both smaller and faster than the best established tool — see the dominance table above. Where mode small wins big on text it does cost 50–100× longer unpacking, and that trade-off is explicit: normal unpacks the code corpus in one second, faster than tar.xz. And fast makes the same corpus 29.5 % smaller than zip -6 in a third of the time.

Can it be made faster? Not by compressing worse, which was a surprise. With compression switched off entirely, fast takes 660 of its 740 ms — the codec is eleven percent of the time. The rest is five sweeps over the data: read, hash, read again, copy into the block, write. The road to a faster mode runs through fewer sweeps, not a weaker codec.

What does it do with JPEG? −15.1 % on 1 143 real photos, lossless and md5-verified; no pixel is touched. But say it straight: it is not new, and hz is worse than the known art. packJPG (2006), Lepton (Dropbox 2016) and brunsli do the same thing and publish 20–22 %.

Did you try LZMA2 at maximum? Yes — 7-Zip -mx=9 with a 512 MiB dictionary and a solid archive, a stronger opponent than xz -9e. hz wins on everything with text (code −11.4 %, logs −29.2 %, enwik8 −13.6 %) and sits 0.05 % below even on archives-of-archives, the one place LZMA2 at maximum was first stronger. 7-Zip packs consistently faster, though.

GPU? AVX-512? The GPU mode is built and measured, and the answer is no on this machine. LZ77 matching gets 2.1–2.7× faster on the graphics card — but packing gets no faster at all, because entropy coding is serial and Amdahl's law wins. The code stays in the repo as a measurement instrument. AVX-512 is in the machine and unused; the gain is unmeasured and may be zero, since nvcm's inner loop is dominated by scattered table lookups.

Does it run on Windows or macOS? Yes, since 2026-11-19 — two ways. The optimized C++ binary is Linux-bound (fork, pipe2, /proc/self/exe; on Windows it runs under WSL2). But the repo now has portable/hzpy.py: pure standard-library Python with no dependencies that packs, unpacks, lists and tests hz archives wherever Python 3.8+ runs. It writes ordinary hz archives restricted to the ra/xz codecs with no transforms — the Linux hz unpacks them bit for bit, and hz pack --portable makes archives that travel the other way. Slower and somewhat larger, on purpose: that is the price of running everywhere.

Is any of this new? No single method. Similarity sorting, deduplication, deflate reconstruction by recipe search, lossless JPEG recoding, context mixing and content-aware filter choice are all long published, and the report has a section crediting each. What is original is the packaging — the parts in one format with one reconstruction guarantee, codec choice settled by probing rather than rule — and the measurement of what the packaging is worth.

Has the code been reviewed by anyone else? Yes, and it gave more than anything else. Ted Lyngmo read the whole C++ codebase and found three real faults: the PNG recipe was not portable across zlib versions, write failures could cause silent data loss since the gzip surface deletes originals, and an unreadable file crashed the process with a core dump. All three are fixed, and his reproduction lives in the repo as a regression test (prov/brister.sh). Fixing the first revealed something worse than reported: a stream of the right length but wrong bytes would have produced a PNG with wrong image data and a correct-looking checksum. Silent corruption is the one thing a packer must never do, and it was there.

The three faults shared a cause worth keeping: hz's own check packs and unpacks on the same machine, in a successful flow, with readable files and free disk. All three lived outside that boundary.

Built as

One C++17 binary (~430 kB): the in-house codecs nvcm and nvjpg are linked in and run as worker processes via /proc/self/exe; zstd, xz and zlib are called as libraries in the same process. The source went English on 2026-11-19 with the Swedish command names kept as permanent aliases, and portable/hzpy.py carries the archives to macOS and Windows. Everything — code, measurement data, both report languages and the scripts that regenerate every table — is at github.com/hugowikstrom/hz.

Downloads

The Linux tool builds from source (make && make install). For macOS and Windows there is the portable edition — pure Python standard library, slower and with somewhat larger archives, but bit-exact and interchangeable with the Linux hz (its --portable mode).

unpack and run ./install.sh, or brew install --formula with hz.rb

unpack and run install.ps1 (needs Python 3.8+ from python.org or the Microsoft Store)

python3 hz-2.0.pyz pack <dir> <archive.nv>

(GPL-2.0-only): the full Linux tool with its codecs and test suite

Checksums: SHA256SUMS. The packages build deterministically — the same commit yields the same bytes.

← All projects