HyperMinHash

Bolts a small MinHash-style signature onto every HyperLogLog register, so a single HLL-family sketch can also estimate Jaccard similarity, intersection size, and union size between two streams, not just distinct-count cardinality.

How it works

Start from a plain HyperLogLog register: the first p bits of an item's hash select one of 2p registers, and the register keeps ρ (rho), the maximum leading-zero-run length seen for that bucket. That alone gives cardinality, exactly like the HyperLogLog page.

HyperMinHash adds one more small field per register: aux, a b-bit hash (b is typically 4–8 bits) of whichever item most recently pushed that register to its current max ρ. When a new item reaches a strictly higher ρ, it becomes the register's new champion and aux is overwritten. When an item ties the current max ρ, the register keeps the smaller of the two aux values: the ordinary MinHash update rule, just scoped to "the items that are this register's current champion" instead of "every item in the set".

Two sketches that both saw the same item route it to the same register with the same ρ and the same aux deterministically. So comparing (ρ, aux) pairs register-by-register between sketch A and sketch B gives a MinHash-like collision signal: registers where both sketches agree are (probably) driven by a shared item, registers that disagree are (probably) not. Averaging that signal across every "touched" register (corrected for the small chance two unrelated items coincidentally produce the same (ρ, aux) pair) estimates the Jaccard similarity. From there, |A∩B| ≈ J × |A∪B|, and |A∪B| itself comes from the ordinary register-wise HLL merge (keep the max ρ per register). All of this stays in O(2^p) space, with no separate MinHash structure needed alongside the cardinality sketch.

Interactive demo

Stream A

True |A|0
Estimate |A|0.0

Stream B

True |B|0
Estimate |B|0.0

Registers: A (top bars) vs. B (bottom bars), match row in the middle

Sketch A registers (top) / Sketch B registers (bottom)middle row: matching (rho & aux agree) touched, no match
Exact Jaccard1.000
Estimated Jaccard1.000
Jaccard abs. error0.000
Exact |A∪B|0
Estimated |A∪B|0.0
Exact |A∩B|0
Estimated |A∩B|0.0
No operations yet. Try adding an item above.

API surface

OperationDescriptionComplexity
new HyperMinHash(p, b)Allocate 2^p registers, each with an 8-bit rho and a b-bit aux MinHash signature.O(2^p) space
add(item): HmhStepUpdate the max-rho register for the item's bucket; on a rho tie, keep the smaller aux (MinHash rule).O(1) time
estimate(): numberCardinality estimate, identical in spirit to HyperLogLog (aux ignored).O(2^p) time
jaccard(other): numberFraction of touched registers whose (rho, aux) pair matches, corrected for false collisions.O(2^p) time
merge(other): HyperMinHashRegister-wise max-rho union of two sketches (keeps the winning, or tie-min, aux).O(2^p) time
unionEstimate(other): numberestimate() of merge(other), the usual HLL union cardinality.O(2^p) time
intersectionEstimate(other): numberjaccard(other) × unionEstimate(other).O(2^p) time