ssdeep: fuzzy hashing explained

Between the cryptographic hash, which shouts «different!» over a single bit, and the perceptual hash, which looks at images through half-closed eyes, sits a third family: fuzzy hashing. It answers a question neither of the others can even phrase: «how alike are these two files, as sequences of bytes?»

The problem it solves

You have two Word documents. One is the draft, the other the signed version, with three lines changed out of eight hundred. SHA-256 tells you they are different files — true and useless. A perceptual hash does not apply: they are not images.

You have two variants of the same malware, recompiled with one function renamed. Ninety-five per cent of the bytes are identical. SHA-256 draws no relation between them, and any catalogue built on exact digests treats them as two unconnected threats.

What is needed here is a fingerprint that changes in proportion to how much the binary content changes. That is precisely what ssdeep does, the most widespread implementation of a technique called CTPHcontext triggered piecewise hashing.

How it works: cut where the content says to cut

The insight is elegant, and it lives entirely in how the file is sliced.

If you sliced it every 4096 bytes, inserting a single character at the start would shift every subsequent boundary by one byte: every slice would change, and the fingerprint would be unrecognisable. That is the alignment problem, and it sinks the idea before it begins.

CTPH cuts differently. It slides a window of a few bytes across the file and computes a rolling hash over it — a value that updates in constant time as the window advances. When that value lands on a predetermined pattern — a «trigger» — the file is cut there. The boundary, in other words, depends not on position but on local content.

Insert a character at the start: the first block changes, its trigger shifts, but every subsequent trigger reappears exactly where it was, because the content that causes it has not changed. From the second block onward, everything is identical.

FIXED-LENGTH CUTTING — one extra byte and it all collapses block 1block 2block 3block 4 1'2'3'4' One byte inserted at the head shifts every boundary: all blocks change, the fingerprint is unrecognisable. CONTENT-TRIGGERED CUTTING (CTPH) — the boundaries hold block 1block 2block 3block 4 triggertriggertrigger 1' changed2 identical 3 identical4 identical The boundary follows local content, not position: only one block changes. Three quarters of the fingerprint survive. Conceptual sketch: ssdeep emits one character per block, yielding two strings comparable by edit distance.
The sliding window looks for a pattern in the bytes. Where it finds one, it cuts. That is why an insertion does not drag every later boundary along with it.

From each block ssdeep derives a single character and concatenates them. The result is a short string that looks something like this:

3072:aBcDeF1gH2iJkL3mN4:aBcD1gH2iJ3mN4

The first part is the block size, which ssdeep picks from the file's length. Then come two fingerprints, computed at block size b and 2b: they allow files of different sizes to be compared without the block choice skewing the result.

Two such strings are compared by edit distance — how many insertions, deletions and substitutions turn one into the other — normalised into a score from 0 to 100.

What Probatio does, and what it does not

This must be stated precisely, because it is where the literature on ssdeep and this software part ways.

Probatio computes a file's ssdeep fingerprint during file analysis, and displays it alongside entropy, magic bytes, strings, executable sections and YARA. It is a datum you can copy, record in a report, or query against external services.

Probatio does not compare two ssdeep fingerprints against each other. There is, today, no ssdeep similarity score in the interface. The comparison is yours to make, with whatever tool you prefer.

Nor does it implement TLSH, the other fuzzy hash widely used in anti-malware work. If you read a comparison attributing it to Probatio, it did not come from here.

Three fingerprints, three questions

SHA-256 «is it the same file?» identical bytes · yes or no resists an adversary seals the evidence ssdeep «how many bytes do they share?» similar bytes · score 0-100 does not resist an adversary clusters the variants pHash «do they look alike?» similar appearance · bit distance images only finds the photo again An edited .docx: SHA-256 says «different», ssdeep says «97», pHash does not apply. A re-compressed photo: SHA-256 says «different», ssdeep says «0», pHash says «96.9%». None of the three substitutes for the other two.
The second example explains the commonest confusion: ssdeep on a re-compressed photo scores zero, because JPEG compression rewrites the bytes from scratch. Fuzzy hashing is not perceptual hashing.

The limits, which are serious

  • It is not cryptographic. Anyone who knows the algorithm can build two files with nearly identical ssdeep fingerprints and wildly different contents. A high score proves nothing against a deliberate adversary.
  • It suffers under compression. Two ZIP archives holding the same document with a minimal edit have uncorrelated ssdeep values: compression rewrites everything. The same goes for JPEG, optimised PDF, executables packed with UPX.
  • The score is not a probability. «72» does not mean the files are 72% the same. It is an index of how many fragments they share, and it is sensitive to block size.
  • It does not say where. A high score does not point to which parts of the file coincide. For that you need a binary comparison, which in Probatio is the diff module.

Using it in practice

  1. To cluster, not to conclude. On a seized volume with ten thousand documents, ssdeep surfaces the near-duplicates and shrinks the work. Then you look at them.
  2. On uncompressed formats. Text, logs, memory dumps, unpacked executables: this is where fuzzy hashing pays. On an optimised PDF, far less.
  3. As an indicator, never as a seal. In a report the ssdeep value goes next to SHA-256, not instead of it.
  4. Together with the binary diff. ssdeep says «these two resemble each other»; the diff says «here is exactly where they differ». The second sentence is the one that belongs in an expert report.

Fuzzy hashing is an investigative tool, not a certifying one. It helps you decide what to open first — which, when the files number in the tens of thousands, is already a great deal.