High praise indeed! Variyam co-develops innovative new data analysis algorithm

The authors presented their new algorithm at the 2022 European Symposium on Algorithms, a premier international conference on algorithm design. Since then, their discovery has been gaining recognition and praise from computer scientists across the field, including world-renowned computer scientist Donald Knuth, author of "The Art of Computer Programming," who is often referred to as "the father of the analysis of algorithms.”

In May 2023, Knuth published his own paper on the algorithm, “The CVM Algorithm for Estimating Distinct Elements in Streams [PDF],” offering extensive praise and even naming it the CVM algorithm in honor of its inventors. Knuth said in addition to “explaining the ideas to just about everybody [he] meets,” he expects the algorithm to become a foundational aspect of computer science in the near future.

"Their algorithm is not only interesting; it is extremely simple,” Knuth said in the paper. “It’s wonderfully suited to teaching students who are learning the basics of computer science. I’m pretty sure that something like this will eventually become a standard textbook topic.”

As Knuth predicted and Variyam hoped, the algorithm has already found a place in computer science courses.

From Knuth’s paper:

Algorithm D (Distinct element estimation). Given an arbitrary data stream A = (a~1~, a~2~, . . . , a~m~) and a buffer size s ≥ 1 as described above, this algorithm returns an unbiased estimate of |A| = |{a~1~, a~2~, . . . , a~m~}|. It uses a buffer B that’s capable of holding up to s ordered pairs (a, u), where a is an element of the stream and u is a real number, 0 ≤ u < 1.

D1. [Initialize.] Set t ← 0, p ← 1, and B ← ∅.
D2. [Done?] Terminate if t = m, returning the estimate |B|/p.
D3. [Input a.] Set t ← t + 1 and a ← a~t~, the next element of the stream.
D4. [Remove a from B.] If B contains the pair (a, u), for any u, delete it.
D5. [Maybe put a in B.] Let u be a uniform deviate, independent of all others (namely a random real number in the range 0 ≤ u < 1). If u ≥ p, go back to step D2. Otherwise, if |B| < s, insert (a, u) into B and go back to step D2.
D6. [Maybe swap a into B.] (At this point u < p and |B| = s.) Let (a', u') be the element of B for which u' is maximum. If u > u', set p ← u. Otherwise replace (a', u') in B by (a, u) and set p ← u'. Then go back to step D2.