Recordinality
Estimates the number of distinct items in a stream by counting "records" in the random permutation induced by hashing each item, a fundamentally different mechanism from HyperLogLog (leading-zero runs) or MinHash/Theta Sketch (minimum-hash tracking).
How it works
Hash every distinct item to a pseudo-uniform value in [0, 1). Because the hash is effectively random, hashing the whole stream this way is equivalent to drawing a uniformly random permutation of the distinct items (the "data streams as random permutations" framing from the 2012 paper). An item is a k-record if fewer than k earlier items in the permutation exceed it. The classical case k=1 is the textbook "new maximum so far" record; general k generalizes this to "one of the k largest values seen so far".
Algorithmically, a k-record is detected by keeping a small table of the k largest hash values seen: if the table isn't full yet, or the new value beats the table's current minimum, it's a record and the table mutates (an insert, or an insert-and-evict). Classical records theory says the number of records after n draws grows like k · log(n/k), so counting records and inverting that growth curve recovers an estimate of n. No leading-zero counting or minimum-value tracking is involved: the count of table mutations is the entire signal.
A pleasant side effect: because the kept table always holds the k largest hashed values seen so far, once the stream ends those items form a uniform random sample of the distinct items in the stream (the paper's "distinct sampling" result), unlike reservoir sampling, repeated items never bias which distinct values are more likely to be retained. Recordinality is also one of the few cardinality estimators with a fully characterized limit distribution (gamma-distributed), rather than just an empirical error bound.
Interactive demo
Error vs. truth: n/a. Approximate standard error at this size: ±0.00%. Add more distinct items to see the estimate converge toward the true count.
The slots below are the kept table: the k largest hash values seen so far, doubling as a uniform random sample of the distinct items in the stream.
API surface
| Operation | Description | Complexity |
|---|---|---|
| new Recordinality(k) | Allocate a kept table that tracks the k largest hash values seen. | O(k) space |
| add(item): RecordinalityStep | Hash the item; detect and apply a k-record mutation if it qualifies. | O(k) time |
| estimate(): number | Unbiased cardinality estimate from the observed record count R. | O(1) time |
| standardErrorFor(n): number | Approximate relative standard error for a stream of true size n. | O(1) time |
| sample(): string[] | The kept table's items: a uniform random sample of distinct items seen. | O(k) time |
| values(): number[] | The kept table's hash values, largest-first. | O(k log k) time |
| clear(): void | Reset the table, record count, and item counter. | O(1) time |
| Recordinality.merge(a, b): Recordinality | Approximate merge of two same-k sketches by replaying one's table into a copy of the other. | O(k^2) time |
References
Original papers
- An Optimal Cardinality Estimation Algorithm Based on Order Statistics and Its Full AnalysisJérémie Lumbroso · 2010
- Data Streams as Random Permutations: the Distinct Element ProblemAhmed Helmi, Jérémie Lumbroso, Conrado Martínez, Alfredo Viola · 2012
Library implementations
Real-world use cases
- Distinct-value sampling: retrieving actual sample elements from a stream alongside a cardinality estimate, not just a count, useful when downstream code wants representative distinct items
- Cardinality estimation with a fully analyzed limit distribution (gamma-distributed error), rather than only empirical/asymptotic error bounds, valuable where rigorous confidence intervals on the estimate matter