SQUISH

SQUISH

A context-mixing file compressor.

Ten statistical models predict every bit, an online-trained logistic mixer fuses their votes, and an arithmetic coder turns the probabilities into bits. It outcompresses zip -9, bzip2 -9, and rar -m5 on 23 of 24 standard-corpus files — shipped as a linkable C library and a CLI.

0.214
compression ratio (315 MB corpus)
23 / 24
files beat zip · bzip2 · rar
10
specialist models mixed per bit
0
dependencies beyond libc / libm

Compression is prediction

If you can estimate the probability of the next bit accurately, an arithmetic coder converts those probabilities into a near-optimal code for free. All the effort goes into one fast, online-learning predictor.

▦ Ten specialist models

Order-0…6 byte contexts, a word model, an auto-detected record model, and an unbounded-window match model each vote on P(bit=1) — so SQUISH behaves like LZ on repetitive data, like PPM on text, and like a record codec on tables, without being told which is which.

∑ Logistic mixing

A tiny one-layer network fuses the ten opinions as a weighted sum of log-odds, with the weight set chosen by context and trained online by gradient descent on real coding loss. Useless models are learned away in a few KB.

⇄ Match & record models

A rolling hash finds the most recent identical context and predicts the history simply repeats — LZ77’s long-range power with an unbounded window, but probabilistically. A vote table detects fixed row lengths, lighting up “cell above” contexts that beat RAR’s filters on spreadsheets.

◈ The model is the format

The decompressor runs the identical predictor in lockstep, so the format needs no dictionaries, tables, or block structure. Every coded block carries a 32-bit checksum; incompressible data falls back to stored blocks, so expansion is bounded.

The pipeline, per bit

order-0..6 byte contexts (7) ┐
word model                   ├─▶ each emits P(bit=1) from an adaptive counter
record model (row length R)  │
match model (6-byte hash)    ┘
                 │
                 ▼   logistic mixer — 11 inputs, 1024 context-selected weight sets
                 ▼   2 chained APM/SSE stages — recalibrate against what happened
                 ▼   carryless 32-bit binary arithmetic coder
              output bits

Full algorithm write-up: SQUISH.md · byte-level spec: docs/FORMAT.md

Results

Silesia + Canterbury + enwik8 — 315 MB across 24 files, every file round-trip verified. Ratio-optimal single-block SQUISH against each rival’s strongest setting.

fileoriginalzip -9bzip2 -9rar -m5xz -9eSQUISHvs best
enwik8100,000,00036,493,32829,008,75826,922,42124,831,64822,461,575+9.5%
silesia/mozilla51,220,48019,032,06717,914,39215,147,51313,376,24014,305,318−6.9%
silesia/webster41,458,70312,114,4128,644,7149,030,4828,368,6726,584,906+21.3%
silesia/nci33,553,4452,998,6181,812,7341,858,8381,449,2721,643,448−13.4%
silesia/samba21,606,4005,406,4464,549,7594,107,4543,739,5243,529,273+5.6%
silesia/dickens10,192,4463,859,2022,799,5203,032,4242,831,2122,300,272+17.8%
silesia/osdb10,085,6843,667,6022,802,7923,237,0632,844,5562,401,772+14.3%
silesia/mr9,970,5643,656,2642,441,2802,775,6642,751,8922,200,359+9.9%
silesia/x-ray8,474,2405,957,3014,051,1124,162,4494,491,2643,732,271+7.9%
silesia/sao7,251,9445,318,1804,940,5245,542,7414,425,6643,968,327+10.3%
silesia/reymont6,627,2021,826,4971,246,2301,444,9331,315,5921,112,376+10.7%
silesia/ooffice6,152,1923,078,3672,862,5262,302,5292,427,2242,412,103−4.8%
silesia/xml5,345,280661,981441,186484,735434,892385,143+11.4%
canterbury/kennedy.xls1,029,744207,125130,28035,65151,86825,479+28.5%
canterbury/ptt5513,21652,78349,75949,34839,86046,231−16.0%
canterbury/plrabn12.txt481,861194,869145,577175,669165,456130,534+10.3%
canterbury/lcet10.txt426,754144,475107,706126,510119,48894,649+12.1%
canterbury/alice29.txt152,08954,32043,20251,26248,52838,687+10.5%
canterbury/asyoulik.txt125,17948,95039,56947,00944,59235,725+9.7%
canterbury/sum38,24012,97612,90911,6849,50011,447−20.5%
canterbury/cp.html24,6038,0697,6248,1567,6526,787+11.0%
canterbury/fields.c11,1503,2123,0393,2023,0322,819+7.0%
canterbury/xargs.14,2271,8301,7621,8601,8121,710+3.0%
canterbury/grammar.lsp3,7211,3161,2831,3591,2921,252+2.4%
TOTAL314,749,364104,800,19084,058,23780,560,95673,780,73267,432,463+8.6%

SQUISH beats zip + bzip2 + rar on 23/24 files, and beats all four (including xz -9e) on 19/24. “vs best” compares SQUISH to the smallest rival output for that file. The trade: ~0.6 MB/s, symmetric, ~150 MB of model memory per active block — SQUISH spends CPU that zip/bzip2/rar don’t and buys ratio with it. Full table: bench/RESULTS.md.

Command-line tool

Everything SQUISH writes is one format: a SQUISH archive. The input may be a file or a whole directory tree — either way the output is an archive, each file its own member.

./squish c bigfile bigfile.sqsh        # compress a file
./squish d bigfile.sqsh restored       # restore it (checksum-verified)
./squish c project project.sqsh        # pack a whole directory tree
./squish l project.sqsh                # list the archive's contents
./squish x project.sqsh src/main.c     # extract one file (or a subtree)
./squish d project.sqsh restored-dir   # recreate the whole tree
./squish -t 0 c bigfile bigfile.sqsh   # compress on all cores
./squish -t 0 -b 4 c big big.sqsh      # ... with 4 MiB blocks (more parallel)

Library

SQUISH is a linkable C library — squish.h + libsquish.so (or squish.dll on Windows). Python needs no wrapper; libsquish.so loads directly with ctypes.

Compress a buffer and back

#include <squish.h>

/* trailing args: threads (0 = all cores, 1 = serial),
 * chunk size (0 = 16 MiB), progress callback + user pointer */
void  *c, *d;  size_t cn, dn;
squish_compress_alloc(data, n, &c, &cn, 1, 0, NULL, NULL);
squish_decompress_alloc(c, cn, &d, &dn, 0, NULL, NULL);   /* integrity-checked */
squish_free(c);  squish_free(d);

Pack a tree, reach one member without inflating the rest

squish_archive_create("project", "project.sqsh", 0, 0, NULL, NULL);
squish_archive *a;
squish_archive_open("project.sqsh", &a);
squish_archive_extract_path(a, "src/main.c", &d, &dn);   /* just this file */
squish_free(d);  squish_archive_close(a);

Link with -lsquish -lm -pthread (or pkg-config --cflags --libs squish after install). Full reference: docs/API.md · examples: example.c, example.py.

Build & install

No dependencies beyond libc / libm.

make            # libsquish.so + squish CLI
make test       # build and run the test suite
make install    # headers, libs, CLI, pkg-config to /usr/local
make dll        # squish.dll + squish.exe via mingw-w64
make windows-dll   # ... via MSVC (cl.exe on PATH)

On Windows without make, run build-windows.bat — it locates Visual Studio automatically and builds squish.dll + squish.exe.

Guarantees & limits