ssdeep: fuzzy hashing explained
Between the cryptographic hash, which shouts «different!» over a single bit, and the perceptual hash, which looks at images through half-closed eyes, sits a third family: fuzzy hashing. It answers a question neither of the others can even phrase: «how alike are these two files, as sequences of bytes?»
The problem it solves
You have two Word documents. One is the draft, the other the signed version, with three lines changed out of eight hundred. SHA-256 tells you they are different files — true and useless. A perceptual hash does not apply: they are not images.
You have two variants of the same malware, recompiled with one function renamed. Ninety-five per cent of the bytes are identical. SHA-256 draws no relation between them, and any catalogue built on exact digests treats them as two unconnected threats.
What is needed here is a fingerprint that changes in proportion to how much the binary content changes. That is precisely what ssdeep does, the most widespread implementation of a technique called CTPH — context triggered piecewise hashing.
How it works: cut where the content says to cut
The insight is elegant, and it lives entirely in how the file is sliced.
If you sliced it every 4096 bytes, inserting a single character at the start would shift every subsequent boundary by one byte: every slice would change, and the fingerprint would be unrecognisable. That is the alignment problem, and it sinks the idea before it begins.
CTPH cuts differently. It slides a window of a few bytes across the file and computes a rolling hash over it — a value that updates in constant time as the window advances. When that value lands on a predetermined pattern — a «trigger» — the file is cut there. The boundary, in other words, depends not on position but on local content.
Insert a character at the start: the first block changes, its trigger shifts, but every subsequent trigger reappears exactly where it was, because the content that causes it has not changed. From the second block onward, everything is identical.
From each block ssdeep derives a single character and concatenates them. The result is a short string that looks something like this:
3072:aBcDeF1gH2iJkL3mN4:aBcD1gH2iJ3mN4
The first part is the block size, which ssdeep picks from the file's length. Then come two fingerprints, computed at block size b and 2b: they allow files of different sizes to be compared without the block choice skewing the result.
Two such strings are compared by edit distance — how many insertions, deletions and substitutions turn one into the other — normalised into a score from 0 to 100.
What Probatio does, and what it does not
This must be stated precisely, because it is where the literature on ssdeep and this software part ways.
Probatio computes a file's ssdeep fingerprint during file analysis, and displays it alongside entropy, magic bytes, strings, executable sections and YARA. It is a datum you can copy, record in a report, or query against external services.
Probatio does not compare two ssdeep fingerprints against each other. There is, today, no ssdeep similarity score in the interface. The comparison is yours to make, with whatever tool you prefer.
Nor does it implement TLSH, the other fuzzy hash widely used in anti-malware work. If you read a comparison attributing it to Probatio, it did not come from here.
Three fingerprints, three questions
The limits, which are serious
- It is not cryptographic. Anyone who knows the algorithm can build two files with nearly identical ssdeep fingerprints and wildly different contents. A high score proves nothing against a deliberate adversary.
- It suffers under compression. Two ZIP archives holding the same document with a minimal edit have uncorrelated ssdeep values: compression rewrites everything. The same goes for JPEG, optimised PDF, executables packed with UPX.
- The score is not a probability. «72» does not mean the files are 72% the same. It is an index of how many fragments they share, and it is sensitive to block size.
- It does not say where. A high score does not point to which parts of the file coincide. For that you need a binary comparison, which in Probatio is the diff module.
Using it in practice
- To cluster, not to conclude. On a seized volume with ten thousand documents, ssdeep surfaces the near-duplicates and shrinks the work. Then you look at them.
- On uncompressed formats. Text, logs, memory dumps, unpacked executables: this is where fuzzy hashing pays. On an optimised PDF, far less.
- As an indicator, never as a seal. In a report the ssdeep value goes next to SHA-256, not instead of it.
- Together with the binary diff. ssdeep says «these two resemble each other»; the diff says «here is exactly where they differ». The second sentence is the one that belongs in an expert report.
Fuzzy hashing is an investigative tool, not a certifying one. It helps you decide what to open first — which, when the files number in the tens of thousands, is already a great deal.