HyperMinHash
Bolts a small MinHash-style signature onto every HyperLogLog register, so a single HLL-family sketch can also estimate Jaccard similarity, intersection size, and union size between two streams, not just distinct-count cardinality.
How it works
Start from a plain HyperLogLog register: the first p bits of an item's hash select one of 2p registers, and the register keeps ρ (rho), the maximum leading-zero-run length seen for that bucket. That alone gives cardinality, exactly like the HyperLogLog page.
HyperMinHash adds one more small field per register: aux, a b-bit hash (b is typically 4–8 bits) of whichever item most recently pushed that register to its current max ρ. When a new item reaches a strictly higher ρ, it becomes the register's new champion and aux is overwritten. When an item ties the current max ρ, the register keeps the smaller of the two aux values: the ordinary MinHash update rule, just scoped to "the items that are this register's current champion" instead of "every item in the set".
Two sketches that both saw the same item route it to the same register with the same ρ and the same aux deterministically. So comparing (ρ, aux) pairs register-by-register between sketch A and sketch B gives a MinHash-like collision signal: registers where both sketches agree are (probably) driven by a shared item, registers that disagree are (probably) not. Averaging that signal across every "touched" register (corrected for the small chance two unrelated items coincidentally produce the same (ρ, aux) pair) estimates the Jaccard similarity. From there, |A∩B| ≈ J × |A∪B|, and |A∪B| itself comes from the ordinary register-wise HLL merge (keep the max ρ per register). All of this stays in O(2^p) space, with no separate MinHash structure needed alongside the cardinality sketch.
Interactive demo
Stream A
Stream B
Registers: A (top bars) vs. B (bottom bars), match row in the middle
API surface
| Operation | Description | Complexity |
|---|---|---|
| new HyperMinHash(p, b) | Allocate 2^p registers, each with an 8-bit rho and a b-bit aux MinHash signature. | O(2^p) space |
| add(item): HmhStep | Update the max-rho register for the item's bucket; on a rho tie, keep the smaller aux (MinHash rule). | O(1) time |
| estimate(): number | Cardinality estimate, identical in spirit to HyperLogLog (aux ignored). | O(2^p) time |
| jaccard(other): number | Fraction of touched registers whose (rho, aux) pair matches, corrected for false collisions. | O(2^p) time |
| merge(other): HyperMinHash | Register-wise max-rho union of two sketches (keeps the winning, or tie-min, aux). | O(2^p) time |
| unionEstimate(other): number | estimate() of merge(other), the usual HLL union cardinality. | O(2^p) time |
| intersectionEstimate(other): number | jaccard(other) × unionEstimate(other). | O(2^p) time |
References
Original papers
- HyperMinHash: MinHash in LogLog SpaceYun William Yu, Griffin M. Weber · 2017
Library implementations
- C (reference impl)yunwilliamyu/hyperminhash
- Goaxiomhq/hyperminhash
- JavaLiveRamp/HyperMinHash-java
- Redis moduleocadaruma/redis-hyperminhash
Real-world use cases
- Streaming Jaccard similarity / set-intersection estimation between two huge event or user-ID streams without materializing either set
- axiomhq/hyperminhash powers similarity estimation between very large sets in Go services at loglog-space cost
- Redis module use for approximate set-similarity queries (A∩B, A∪B, Jaccard) directly on HLL-style sketches