Jekyll2023-02-06T18:25:47+00:00https://alinush.github.io//feed.xmlAlin TomescuTBD
Alin TomescuWhy you should probably never sort your Merkle tree’s leaves2023-02-05T00:00:00+00:002023-02-05T00:00:00+00:00https://alinush.github.io//2023/02/05/Why-you-should-probably-never-sort-your-Merkle-trees-leaves<p class="info"><strong>tl;dr:</strong> …because (1) they are only secure when the tree is correctly-computed (e.g., secure with BFT consensus, but <strong>insecure</strong> in single-server transparency logs), (2) you cannot efficiently insert or delete leaves, and (3) they have worse proof sizes. What does that mean? Never implement one. Stick to <a href="#probably-use-a-merkle-trie">Merkle tries</a> (a.k.a., Merkle prefix trees). Or, if you are a masochist and like to deal with rotations, stick to balanced binary search trees.</p>
<!--more-->
<p hidden="">$$
\def\Adv{\mathcal{A}}
\def\Badv{\mathcal{B}}
\def\vect#1{\mathbf{#1}}
$$</p>
<hr />
<h2 id="problem-statement">Problem statement</h2>
<p>Non-membership proofs in <a href="/2021/02/25/what-is-a-merkle-tree.html">Merkle trees</a> are surprisingly elusive to many people.
The problem statement is very simple:</p>
<p style="margin-left: .3em; border-left: .25em solid black; padding-left: .5em;">
Suppose you have a <strong>server</strong> who wants to authenticate elements of a set $S$ to a <strong>client</strong> without ever sending the whole set to this client.
(For simplicity, let’s assume this is a set of numbers.)
<br /><br />
Specifically, the server first computes a succinct <strong>authentication digest</strong> of the set, denoted by $d$, and sends $d$ to the client.
Then, the server is able to prove either <em>membership</em> or <em>non-membership</em> of an element in the set by sending a succinct <strong>proof</strong> to the <strong>client</strong> which the client can efficiently verify with respect to the digest $d$.
<br /><br />
Design a Merkle tree-based solution for this problem.
</p>
<p>The most popular solution to this problem seems to be to build a Merkle tree whose leaves are sorted.
This, unfortunately, is a rather sub-optimal solution, both from a security and a complexity point of view.</p>
<p>In this blog post, I hope to dispel the myth of the effectiveness of this <em>sorted-leaves</em> Merkle tree scheme.</p>
<h2 id="warm-up-proving-membership-in-merkle-trees">Warm-up: proving membership in Merkle trees</h2>
<p>Recall that it, if we only need to prove membership, it is very easy to solve the problem by building <a href="/2021/02/25/what-is-a-merkle-tree.html">a Merkle tree</a> over all elements in the set and letting the digest be the Merkle root.</p>
<p>For example, here’s how this would look for a particular choice of set $S$. (Original slides <a href="https://docs.google.com/presentation/d/1pbQLuXcI6WjyuNd787ty3U6Qbon41f91e_ixDmgpUgw/edit?usp=sharing">here</a>.)</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-unsorted.png" /></div>
<p>Then, a membership proof would be a <strong>Merkle sibling path</strong> to the proved element’s leaf (i.e., the nodes in <span style="color: #b58900;"><b>yellow</b></span>):</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-memb.png" /></div>
<p>The client can easily verify the proof by computing the hashes along the path from the leaf to the root, and checking that it obtains the same root hash as it has stored locally:</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-memb-verify.png" /></div>
<h2 id="sort-the-leaves-to-add-support-for-non-membership">Sort the leaves to add support for non-membership?</h2>
<p>It seems that many people believe sorting the leaves is the right approach to enable <strong>non</strong>-membership proofs.</p>
<p>This blog post will argue, from three different perspectives, why this is a sub-optimal choice.</p>
<p>Okay! Let say that, instead of the solution <a href="#warm-up-proving-membership-in-merkle-trees">from above</a>, the server does indeed first <strong>sort</strong> the set $S$ as $[2, 3, 6, 7, 9, 11, 13, 14]$ and then computes the Merkle tree:</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-sorted.png" /></div>
<p>Clearly, the server can still prove membership as before: just give a Merkle sibling path to the proved element’s leaf.</p>
<p>But, now, it is also possible to prove non-membership of an element.</p>
<p>For example, we can prove non-membership of $8$ by showing that <em>(a)</em> both $7$ and $9$ are in the tree <strong>and</strong> <em>(b)</em> that they are <strong>adjacent</strong>.
This implies there’s no room where 8 could fit.
Therefore, 8 cannot be in the tree:</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-non-memb.png" /></div>
<p>In other words, the two membership proofs for the adjacent leaves of $7$ and $9$ constitute a <strong>non-membership proof</strong> for 8, which would have to be placed between them (but cannot be since “there’s no room”).</p>
<h2 id="problem-1-security">Problem 1: Security</h2>
<p>Can you spot the security issue? It’s a bit subtle and many people miss it…</p>
<p>Here it is: this scheme is secure <strong>only if</strong> the server correctly computes the Merkle tree over the sorted leaves.</p>
<p>Otherwise, if the server is malicious, it can re-order the leaves and pretend than an element $e$ is <strong>both in</strong> the set <strong>and not in</strong> the set.</p>
<p>For example, the malicious server could compute the tree as follows:</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-incorrectly.png" /></div>
<p>Note that the malicious server left 7 adjacent to 9, so that it can still give <em>what appears to be</em> a valid non-membership proof for 8:</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-attack-non-memb.png" /></div>
<p>At the same time, note that the malicious server inserted a leaf for 8 somewhere else.
As a result, the server can still give <em>what appears to be</em> a valid membership proof for 8:</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-attack-memb.png" /></div>
<p class="error">This, of course, is very bad: the server was able to prove two <strong>inconsistent</strong> statements about the membership of 8 in the digested set. Put differently, it clearly cannot be that 8 is both in $S$ and not in $S$ at the same time. Therefore, the sorted-leaves Merkle tree is insecure when the server cannot be trusted to produce correct digests (and we’ll define security <a href="#nonmembership-soundness-definitions">below</a>).</p>
<p>In other words, this type of attack completely ruins security: it makes any proof <em>meaningless</em> to the client (e.g., proof that $8\in S$), since it could easily be followed by a contradicting proof (e.g., a contradicting proof that $8\notin S$).</p>
<h3 id="when-would-the-sorted-leaves-merkle-tree-be-secure">When would the sorted-leaves Merkle tree be secure?</h3>
<p>Not all hope is lost. In some settings, it can be reasonable to assume the digest (i.e., Merkle root) was produced correctly.</p>
<p>For example, in distributed consensus settings (a.k.a., in “blockchains”), there is no single server that dictates what the Merkle root of the data is.
Instead, all $n = 3f+1$ servers try to compute the same <em>correct</em> root and vote on it.
Servers who deviate from the correct root are ignored and consensus is reached on the correct one by a subset of $2f+1$ honest servers.</p>
<p>Therefore, in this setting, it is okay to rely on the sorted-leaves Merkle tree construction.
(I’ll still argue <a href="#problem-2-insertions-and-deletions">here</a> and <a href="#problem-3-proof-size-is-sub-optimal">here</a> why you shouldn’t, but from different perspectives.)</p>
<p>Other harmless settings include single-client data outsourcing, where a client sorts & Merkle hashes his own data correctly, and transfers everything but the Merkle root to a malicious server.</p>
<p>Since the client has computed the correct root on his own, the client can rely on the server’s (non)membership proofs.</p>
<p class="error">One thing worth emphasizing is that ad-hoc fixes to the problem of a potentially-incorrect digest are not worth it, especially since one can get a construction that needs no fixing from, e.g., a <a href="#probably-use-a-merkle-trie">Merkle trie</a>. Specifically, it is not worth it to require the server to prove that it correctly sorted the leaves (e.g., via a SNARK). Also, it is not worth it to rely on fraud proofs when one can have provably-correct behavior all the time. Lastly, it is not worth it to probabilistically audit the data structure to see if you can find two incorrectly-sorted leaves. None of these approaches are worth it because there exist more secure Merkle tree constructions like Merkle tries. Plus, these constructions are easier to update and have smaller proof sizes!</p>
<h3 id="nonmembership-soundness-definitions">(Non)membership soundness definitions</h3>
<p>We can formalize the setting in which authenticated set constructions (like the sorted-leaves Merkle tree) are secure.</p>
<p>Specifically, we can define a notion of <em>weak (non)membership soundness</em> that captures the idea that the malicious server must compute the digest correctly:</p>
<p class="info">An authenticated set scheme has <strong>weak (non)membership soundness</strong> if for all (polynomial-time) adversaries $A$, the probability that $A$ outputs a set $S$, an element $e$, and two proofs $\pi$ & $\pi’$ such that, letting $d$ be the (correct) digest of $S$, $\pi$ verifies as a valid membership proof for $e$ (w.r.t. $d$) while $\pi’$ also verifies as a valid <strong>non</strong>-membership proof for $e$ (w.r.t. $d$), is negligible in the security parameter of the scheme.</p>
<p>Notice that the adversary outputs a set of elements from which the correct digest $d$ is computed.</p>
<p>In fact, there is a long line of academic literature on 2-party and 3-party authenticated data structures that rely on this type of weaker soundness definitions (see Papamanthou’s PhD thesis<sup id="fnref:Papa11" role="doc-noteref"><a href="#fn:Papa11" class="footnote" rel="footnote">1</a></sup> for a survey).</p>
<p>Unfortunately, many applications today inherently rely on untrusted publishers who can compute malicious digests of their data.</p>
<p>For example, in key transparency logs such as Certificate Transparency (CT), log servers can present any digest to new clients joining the system. Therefore, in this setting, authenticated data structures (whether sets or not), must satisfy a <strong>stronger</strong> notion of security which allows the adversary to construct the digest maliciously.</p>
<p>In fact, such a <strong>stronger</strong> notion simply requires that the adversary output the digest $d$ directly, which gives the adversary freedom to construct an incorrect one as in <a href="#problem-1-security">our attack above</a>:</p>
<p class="info">An authenticated set scheme has <strong>strong (non)membership soundness</strong> if for all (polynomial-time) adversaries $A$, the probability that $A$ outputs a digest $d$, an element $e$, and two proofs $\pi$ & $\pi’$ such that $\pi$ verifies as a valid membership proof for $e$ (w.r.t. $d$) while $\pi’$ also verifies as a valid <strong>non</strong>-membership proof for $e$ (w.r.t. $d$), is negligible in the security parameter of the scheme.</p>
<p>The moral of the story is to pick a Merkle construction that has this stronger notion of security, unless you are sure that your setting allows for the weaker notion <em>and</em> you stand to benefit from relaxing the security (e.g., perhaps because you get a faster construction).
A good example of this is the <a href="/2020/05/06/kzg-polynomial-commitments.html">KZG</a>-based authenticated dictionary from Ethereum Research<sup id="fnref:Feis20Multi" role="doc-noteref"><a href="#fn:Feis20Multi" class="footnote" rel="footnote">2</a></sup> which has <strong>weak soundness</strong> (as would be defined for dictionaries), but that’s okay since their consensus setting can accommodate it.</p>
<h2 id="problem-2-insertions-and-deletions">Problem 2: Insertions and deletions</h2>
<p>This one is much easier to explain.</p>
<p>Imagine you want to add a new element in your sorted-leaves Merkle tree of size 8.</p>
<p>What if it is smaller than everything else and has to be inserted as the first leaf of the tree?</p>
<p>Then, you would have to completely rehash the entire tree to incorporate this new leaf! This would take $O(n)$ work in a tree of $n$ leaves.</p>
<p>The same problem arises if you’d like to remove the first leaf.</p>
<p class="info">To deal with the slowness of <strong>insertions</strong>, one can take an amortized approach and maintain a <strong>forest</strong> of sorted-leaves Merkle trees, where (1) new leaves are appended to the right of the forest as their own size-1 trees and (2) trees of the same size $2^i$ for any $i \ge 0$ get “merged” together by merge-sorting their leaves and rehashing. One can show this approach has $O(\log{n})$ <strong>amortized</strong> insertion cost. However, such amortized approaches still suffer from $O(n)$ worst-case times and must be de-amortized to bring the worst-case cost down to the amortized cost<sup id="fnref:Eric15" role="doc-noteref"><a href="#fn:Eric15" class="footnote" rel="footnote">3</a></sup>.</p>
<p class="info">On the other hand, dealing with <strong>deletions</strong> can be easier. Specifically, if you do not care about wasted space, then deletions can be done faster by simply marking the leaf as “removed” and trying to garbage-collect as many empty subtrees as you can. Nonetheless, in the worst case, the storage complexity of an $n$-leaf Merkle tree after $O(n)$ deletes remains $O(n)$ (e.g., imagine deleting every even-numbered leaf).</p>
<h2 id="problem-3-proof-size-is-sub-optimal">Problem 3: Proof size is sub-optimal</h2>
<p>The other problem with the sorted leaves construction is that <strong>two</strong> Merkle paths must be given as a non-membership proof.</p>
<p>In the best-case, this can be exactly $\log{n}-1$ hashes, but in the worst case this can be as much as $2\log{n}-2$ hashes (e.g., when one proof is in the left subtree and the other proof is in the right subtree).</p>
<p>This is not so great if proof size is a concern.
It is also not so great when the Merkle tree is stored on disk since it can double the proof reading I/O cost.</p>
<p>Furthermore, actually achieving the best-case proof size complexity in an implementation can be tricky: the developer must efficiently batch the fetching of the two Merkle proofs from disk or memory, taking care never to fetch the same sibling hash twice (or waste I/O).</p>
<h2 id="probably-use-a-merkle-trie">Probably use a Merkle trie</h2>
<p>This deserves its own post, but here are the key reasons <strong>you should probably use a Merkle trie</strong>:</p>
<ul>
<li>Tries are an intuitive data structure</li>
<li>Tries do not require rotations to keep the tree well-balanced</li>
<li>Merkle tries offer <strong>strong (non)membership soundness</strong></li>
<li>Merkle tries are relatively-easy to implement</li>
</ul>
<p>There are of course some disadvantages too, but I find them negligible:</p>
<ul>
<li>Tries require a bit more hash computations to determine the path of an element in the tree (during insertion, updates, proof verification, etc.)</li>
<li>Tries have some tricky edge-cases when implementing (e.g., inserting two elements whose first $k$ bits collide in an empty trie)</li>
<li>Tries have some tricky edge-cases for proving non-membership
<ul>
<li>e.g., when proving non-membership of $e$, either a leaf exists along $e$’s path but it’s for the wrong element, or no leaf exists at all</li>
<li>This edge case arises in the simplest implementation of tries, which do not ensure the tree is <strong>full</strong> (i.e., <em>fullness</em> means every node other than the leaves has two children)</li>
</ul>
</li>
<li>Tries are vulnerable to adversarial insertions: an adversary can search for a key whose insertion depth will be very large
<ul>
<li>However, to achieve depth $k$, the adversary will have to compute $2^k$ hashes, which gets expensive quickly.</li>
</ul>
</li>
</ul>
<p class="info">In fact, some folks argue that the best trie implementation is via <a href="https://cr.yp.to/critbit.html">critbit trees</a><sup id="fnref:alnoki" role="doc-noteref"><a href="#fn:alnoki" class="footnote" rel="footnote">4</a></sup>.
Unfortunately, I do not know enough about their benefits, especially when Merkleized, but this is probably very much worth exploring.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Hopefully, this post gave you enough context on the problems of this popular sorted-leaves Merkle tree construction.</p>
<p>This leaves me wondering: are there any advantages to sorted-leaves Merkle trees?</p>
<p>The only advantage I see is that MHTs with sorted leaves are easy to describe: just sort the leaves, Merkleize them and prove non-membership of an element $e$ by revealing the two paths to the adjacent leaves that exclude $e$.</p>
<p>However, just because they are easy to describe does <strong>not</strong> mean they are easy to understand.</p>
<p>At least, from the questions and answers I see online, and from conversations with researchers and other engineers, their security caveats are <strong>not</strong> well understood.</p>
<p>First, <a href="https://crypto.stackexchange.com/a/31955/21296">my own answer on StackExchange</a> makes an unfortunate use of the “sorted Merkle tree” terminology to refer to either a binary search tree<sup id="fnref:bst-def" role="doc-noteref"><a href="#fn:bst-def" class="footnote" rel="footnote">5</a></sup>, a trie, or a Sparse Merkle tree (SMT), which actually all have <strong>strong (non)membership soundness</strong>. Even worse, tries and SMTs are not really sorted, since data is typically hashed before being mapped into the trie.</p>
<p>Another <a href="https://crypto.stackexchange.com/questions/83289/proof-of-membership-and-non-membership-in-merkle-tree-hash-tree/83291#83291">StackExchange answer</a> seems to perpetuate the myth that all you need for non-membership security is to sort the leaves, without paying attention to the <strong>weak (non)membership soundness</strong> guarantees of such a construction.</p>
<p>The answer quotes <a href="https://gist.github.com/chris-belcher/eb9abe417d74a7b5f20aabe6bff10de0">this post</a>, where a sorted-leaves Merkle tree solution is described to solve a non-membership problem like <a href="#problem-statement">the one in the intro</a>.
Unfortunately, the answer discards the nuance of the quoted post: there, the original author realized that the leaves could be incorrectly-sorted & resorted to fraud proofs to catch such misbehaviour; i.e., if someone detects a mis-ordered tree, they can easily prove it with two Merkle paths to the out-of-order leaves.</p>
<p>Yet a <strong>much easier</strong> and <strong>cheaper</strong> solution would have been to use an authenticated set with <strong>strong (non)membership soundness</strong> as defined <a href="#nonmembership-soundness-definitions">above</a> (e.g., a <a href="#probably-use-a-merkle-trie">Merkle trie</a>).
This would have simplified the higher-level protocol, since it would have removed the need for fraud proofs, which are clearly less desirable when one can have provably-correct behavior all the time.</p>
<p>Oh well, we live and learn. <strong>Don’t sort your Merkle tree’s leaves</strong>, okay? Use a <a href="#probably-use-a-merkle-trie">Merkle trie</a>.</p>
<p>And, if you somehow find a reason to sort your leaves, please let me know what were the advantages of doing it.
Don’t forget to compare to more secure solutions such as <a href="#probably-use-a-merkle-trie">Merkle tries</a>, which have <strong>strong (non)membership soundness</strong>.</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:Papa11" role="doc-endnote">
<p><strong>Cryptography for Efficiency: New Directions in Authenticated Data Structures</strong>, by Charalampos Papamanthou, 2011, <a href="https://user.eng.umd.edu/~cpap/published/theses/cpap-phd.pdf">[URL]</a> <a href="#fnref:Papa11" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Feis20Multi" role="doc-endnote">
<p><strong>Multi-layer hashmaps for state storage</strong>, by Dankrad Feist, 2020, <a href="https://ethresear.ch/t/multi-layer-hashmaps-for-state-storage/7211/">[URL]</a> <a href="#fnref:Feis20Multi" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Eric15" role="doc-endnote">
<p><strong>Static-to-dynamic tranformations</strong>, by Jeff Erickson, 2015, <a href="http://jeffe.cs.illinois.edu/teaching/datastructures/notes/01-statictodynamic.pdf">[URL]</a> <a href="#fnref:Eric15" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:alnoki" role="doc-endnote">
<p>Shoutout to Alnoki from <a href="https://www.econialabs.com/">Econia Labs</a> who brought crit-bit trees to my attention. <a href="#fnref:alnoki" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:bst-def" role="doc-endnote">
<p>Note that a <a href="https://en.wikipedia.org/wiki/Binary_search_tree">binary-search tree (BST)</a> is a tree where all left descendants of a node are smaller than that node & all right descendants of a node are greater than that node. Importantly, trees with sorted leaves are not conceptualized as binary search trees, since their data is stored in the leaves, not in the internal nodes. <a href="#fnref:bst-def" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Alin Tomescutl;dr: …because (1) they are only secure when the tree is correctly-computed (e.g., secure with BFT consensus, but insecure in single-server transparency logs), (2) you cannot efficiently insert or delete leaves, and (3) they have worse proof sizes. What does that mean? Never implement one. Stick to Merkle tries (a.k.a., Merkle prefix trees). Or, if you are a masochist and like to deal with rotations, stick to balanced binary search trees.Pairings or bilinear maps2022-12-31T20:45:59+00:002022-12-31T20:45:59+00:00https://alinush.github.io//2022/12/31/pairings-or-bilinear-maps<!-- TODO: Add example of pairing (insecure). -->
<p class="info"><strong>tl;dr:</strong> <em>Pairings</em>, or <em>bilinear maps</em>, are a very powerful mathematical tool for cryptography.
Pairings gave us our most succinct zero-knowledge proofs<sup id="fnref:GGPR12e" role="doc-noteref"><a href="#fn:GGPR12e" class="footnote" rel="footnote">1</a></sup>$^,$<sup id="fnref:PGHR13e" role="doc-noteref"><a href="#fn:PGHR13e" class="footnote" rel="footnote">2</a></sup>$^,$<sup id="fnref:Grot16" role="doc-noteref"><a href="#fn:Grot16" class="footnote" rel="footnote">3</a></sup>, our most efficient threshold signatures<sup id="fnref:BLS01" role="doc-noteref"><a href="#fn:BLS01" class="footnote" rel="footnote">4</a></sup>, our first usable identity-based encryption (IBE)<sup id="fnref:BF01" role="doc-noteref"><a href="#fn:BF01" class="footnote" rel="footnote">5</a></sup> scheme, and many other efficient cryptosystems<sup id="fnref:KZG10" role="doc-noteref"><a href="#fn:KZG10" class="footnote" rel="footnote">6</a></sup>.
In this post, I’ll teach you a little about the properties of pairings, their cryptographic applications and their fascinating history.
In fact, by the end of this post, <a href="#history">some of you might want to spend a year or two in jail</a>.</p>
<p class="warning"><strong>Twitter correction:</strong> The <a href="https://twitter.com/alinush407/status/1612518576862408705">original tweet announcing this blog post</a> stated that <em>“<strong>S</strong>NARKs would not be possible without [pairings]”</em>, with the highlighted <strong>S</strong> meant to emphasize the “succinctness” of such SNARKs. However, <a href="#acknowledgements">thanks to several folks on Twitter</a>, I realized this is <strong>not</strong> exactly true and depends on what one means by “succinct.” Specifically, “succinct” SNARKs, in the <em>polylogarithmic proof size</em> sense defined by Gentry and Wichs<sup id="fnref:GW10" role="doc-noteref"><a href="#fn:GW10" class="footnote" rel="footnote">7</a></sup>, exist from a plethora of assumptions, including discrete log<sup id="fnref:BCCplus16" role="doc-noteref"><a href="#fn:BCCplus16" class="footnote" rel="footnote">8</a></sup> or random oracles<sup id="fnref:Mica98" role="doc-noteref"><a href="#fn:Mica98" class="footnote" rel="footnote">9</a></sup>. Furthermore, “succinct” SNARKs, in the sense of $O(1)$ group elements proof size, exist from RSA assumptions too<sup id="fnref:LM18" role="doc-noteref"><a href="#fn:LM18" class="footnote" rel="footnote">10</a></sup>. What pairings do give us, currently, are SNARKs with the smallest, concrete proof sizes (i.e., in # of bytes).</p>
<p hidden="">$$
\def\idt{\mathbb{1}_{\Gr_T}}
\def\msk{\mathsf{msk}}
\def\dsk{\mathsf{dsk}}
\def\mpk{\mathsf{mpk}}
$$</p>
<h2 id="preliminaries">Preliminaries</h2>
<ul>
<li>You are familiar with cyclic groups of prime order (e.g., elliptic curves)</li>
<li>Let \(\idt\) denote the identity element of the group $\Gr_T$</li>
<li>Let $x \randget S$ denote randomly sampling an element $x$ from a set $S$</li>
<li>Recall that $\langle g \rangle = \Gr$ denotes $g$ being a generator of the group $\Gr$</li>
</ul>
<h2 id="definition-of-a-pairing">Definition of a pairing</h2>
<p>A <em>pairing</em>, also known as a <em>bilinear map</em>, is a function $e : \Gr_1 \times \Gr_2 \rightarrow \Gr_T$ between three groups $\Gr_1, \Gr_2$ and $\Gr_T$ of prime order $p$, with generators $g_1 = \langle \Gr_1 \rangle, g_2 = \langle \Gr_2 \rangle$ and $g_T = \langle \Gr_T \rangle$, respectively.</p>
<p>When $\Gr_1 = \Gr_2$, the pairing is called <strong>symmetric</strong>. Otherwise, it is <strong>asymmetric</strong>.</p>
<p>Most importantly, a pairing has two useful properties for cryptography: <em>bilinearity</em> and <em>non-degeneracy</em>.
<!--more--></p>
<h3 id="bilinearity">Bilinearity</h3>
<p><em>Bilinearity</em> requires that, for all $u\in\Gr_1$, $v\in\Gr_2$, and $a,b\in\Zp$:</p>
\[e(u^a, v^b) = e(u, v)^{ab}\]
<p class="warning">For cryptography purposes, this is the <strong>coolest</strong> property. For example, this is what enables useful applications like <a href="#tripartite-diffie-hellman">tripartite Diffie-Hellman</a>.</p>
<h3 id="non-degeneracy">Non-degeneracy</h3>
<p><em>Non-degeneracy</em> requires that:</p>
\[e(g_1, g_2) \ne \idt\]
<p class="info"><strong>Why this property?</strong> We want non-degeneracy because, without it, it is simple (and useless) to define a (degenerate) bilinear map that, for every input, returns $\idt$. Such a map would satisfy bilinearity, but would be completely useless.</p>
<h3 id="efficiency">Efficiency</h3>
<p><em>Efficiency</em> requires that there exists a polynomial-time algorithm in the size of a group element (i.e.; in $\lambda = \log_2{p}$) that can be used to evaluate the pairing $e$ on any inputs.</p>
<details>
<summary><b>Why this requirement?</b> It precludes trivial-but-computationally-intractable pairings. <i>(Click to expand.)</i></summary>
<p style="margin-left: .3em; border-left: .15em solid black; padding-left: .5em;">
For example, let $r$ be a random element in $\Gr_T$.
First, the pairing is defined so that $e(g_1, g_2) = r$.
This way, the pairing satisfies <em>non-degeneracy</em>.
<br /><br />
Second, given $(u,v)\in \Gr_1 \times \Gr_2$, an algorithm could spend exponential time $O(2^\lambda)$ to compute the discrete logs $x = \log_{g_1}{(u)}$ and $y = \log_{g_2}{(v)}$ and return $e(u, v) = e(g_1^x, g_2^y) = r^{xy}$.
This way, the pairing satisfies <em>bilinearity</em> because:
<br /><br />
\begin{align}
e(u^a, v^b)
&= e\left((g_1^x)^a, (g_2^y)^b\right)\\\
&= e\left(g_1^{(ax)}, g_2^{(by)}\right)\\\
&= r^{(ax)\cdot (by)}\\\
&= \left(r^{xy}\right)^{ab}\\\
&= e(u, v)^{ab}
\end{align}
</p>
</details>
<h2 id="history">History</h2>
<p class="warning">This is my limited historical understanding of pairings, mostly informed from <a href="https://www.youtube.com/watch?v=1RwkqZ6JNeo">Dan Boneh’s account in this video</a> and from my own research into the relevant literature. Please email me if you are aware of more history and I can try to incorporate it.</p>
<h3 id="a-mathematician-in-prison">A mathematician in prison</h3>
<p>The history of (cryptographic) pairings begins with a mathematician named <strong>André Weil</strong><sup id="fnref:Wiki22Weil" role="doc-noteref"><a href="#fn:Wiki22Weil" class="footnote" rel="footnote">11</a></sup> who, during World War II, is sent to jail for refusing to serve in the French army.
There, Weil, <em>“managed to convince the liberal-minded prison director to put [him] in an individual cell where [he] was allowed to keep [..] a pen, ink, and paper.”</em></p>
<p>Weil used his newly-acquired tools to define a pairing across two elliptic curve groups.
<strong>However</strong>, what was <strong>very odd</strong> at that time was that Weil put in a lot of effort to make sure his definition of a pairing is <em>computable</em>.
And this extra effort is what made today’s pairing-based cryptography possible<sup id="fnref:danboneh-shimuranote" role="doc-noteref"><a href="#fn:danboneh-shimuranote" class="footnote" rel="footnote">12</a></sup>.</p>
<h3 id="go-to-prison-not-to-university">Go to prison, not to university?</h3>
<p>Funnily, Weil’s time in jail was so productive that he began wondering if he should spend a few months there every year.
Even better, Weil contemplated if he should <strong>recommend to the relevant authorities that every mathematician spend some amount of time in jail.</strong>
Weil writes:</p>
<blockquote>
<p>I’m beginning to think that nothing is more conducive to the abstract sciences than prison.</p>
<p>[…]</p>
<p>My mathematics work is proceeding beyond my wildest hopes, and I am even a bit worried - if it’s only in prison that I work so well, will I have to arrange to spend two or three months locked up every year?</p>
<p>In the meantime, I am contemplating writing a report to the proper authorities, as follows: <em>“To the Director of Scientific Research: Having recently been in a position to discover through personal experience the considerable advantages afforded to pure and disinterested research by a stay in the establishments of the Penitentiary System, I take the liberty of, etc. etc.”</em></p>
</blockquote>
<p>You can read all of this and more in his fascinating autobiography, written from his perspective as a mathematician<sup id="fnref:Weil92" role="doc-noteref"><a href="#fn:Weil92" class="footnote" rel="footnote">13</a></sup>.</p>
<h3 id="from-breaking-cryptography-to-building-cryptography">From breaking cryptography to building cryptography</h3>
<p>Weil’s work was the foundation.
But three more developments were needed for pairing-based cryptography to rise.</p>
<h4 id="first-development-millers-algorithm">First development: Miller’s algorithm</h4>
<p>In 1985, <strong>Victor Miller</strong> writes up a manuscript showing that Weil’s pairing, which actually involves evaluating a polynomial of exponential degree, can in fact be computed efficiently in polynomial time<sup id="fnref:Mill86Short" role="doc-noteref"><a href="#fn:Mill86Short" class="footnote" rel="footnote">14</a></sup>.</p>
<p>In December 1984, Miller gave a talk at IBM about elliptic curve cryptography where he claimed that elliptic curve discrete logs were more difficult to compute than ordinary discrete logs over finite fields<sup id="fnref:miller-talk" role="doc-noteref"><a href="#fn:miller-talk" class="footnote" rel="footnote">15</a></sup>.
Miller was challenged by Manuel Blum, who was in the audience, to back up this claim by giving a <a href="https://en.wikipedia.org/wiki/Reduction_(complexity)">reduction</a>: i.e., showing that an algorithm $B$ for solving discrete log on elliptic curves can be efficiently turned into another algorithm $A$ for solving discrete logs in finite fields.
Such a reduction would imply the problem solved by $B$ (i.e., computing elliptic curve discrete logs) is at least as hard, if not harder, than $A$’s problem (i.e., computing finite field discrete logs).</p>
<p>Miller set out to find a reduction by thinking about the only thing that related an elliptic curve group and a finite field: the Weil pairing.
Funnily, what he quickly realized was that, although the Weil pairing gives a reduction, it’s in the opposite direction: i.e., it turned out an algorithm $A$ for discrete log in finite fields could be efficiently turned into an algorithm $B$ for discrete logs in elliptic curves with the help of the Weil pairing.
This “unwanted” reduction is easy to see.
Since $e(g^a, g) = e(g,g)^a$, solving discrete log on the elliptic curve element $g_a\in \Gr$ is just a matter of solving it on $e(g,g)^a\in \Gr_T$, which is actually a multiplicative subgroup of a finite field (see <a href="#how-do-pairings-actually-work">the inner details of pairings</a>).</p>
<p>This almost showed the opposite of what Miller sought to prove, potentially destroying elliptic curve cryptography, but fortunately the degree of the extension field that the Weil pairing mapped into was too large, making this “unwanted” reduction inefficient and thus not a reduction after all.</p>
<p>This whole affair got Miller interested in seeing if the Weil pairing could be computed efficiently, which led to the discovery of his algorithm.
Interestingly, he submitted this manuscript to FOCS, a top theoretical computer science conference, but the paper got rejected and would not be published until much later in the Journal of Cryptology (according to Miller)<sup id="fnref:alin-where" role="doc-noteref"><a href="#fn:alin-where" class="footnote" rel="footnote">16</a></sup>.</p>
<h4 id="second-development-mov-attack">Second development: MOV attack</h4>
<p>In 1991, <strong>Menezes, Vanstone and Okamoto</strong><sup id="fnref:MVO91" role="doc-noteref"><a href="#fn:MVO91" class="footnote" rel="footnote">17</a></sup> leverage Miller’s efficient algorithm for evaluating the Weil pairing to break the discrete logarithm assumption on certain elliptic curves <strong>in sub-exponential time</strong>.
This was quite amazing since, at that time, no sub-exponential time algorithms were known for elliptic curves.</p>
<p class="info">Their attack, called the <em>MOV attack</em>, mapped an elliptic curve discrete logarithm challenge $g^a\in \Gr$ into a <strong>target group</strong> as $e(g^a, g)=e(g,g)^a \in \Gr_T$ using the pairing.
Since the target group was a subgroup of a finite field $\mathbb{F}_q^{k}$, this allowed the use of faster, sub-exponential time algorithms for computing the discrete log on $e(g,g)^a$.</p>
<h4 id="third-development-jouxs-tripartite-diffie-hellman">Third development: Joux’s tripartite Diffie-Hellman</h4>
<p>So far, pairings only seemed useful for <strong>cryptanalysis.</strong>
No one knew how to use them for building (instead of breaking) cryptography.</p>
<p>This changed in 2000, when <strong>Joux</strong><sup id="fnref:Joux00" role="doc-noteref"><a href="#fn:Joux00" class="footnote" rel="footnote">18</a></sup> used pairings to implement a 1-round key-exchange protocol between three parties, or <a href="#tripartite-diffie-hellman">tripartite Diffie-Hellman</a>.
Previously, such 1-round protocols were only known between two parties while three parties required 2 rounds.</p>
<p>From there, an abundance of new, efficient cryptography started pouring over:</p>
<ul>
<li>BLS (short) signatures<sup id="fnref:BLS01:1" role="doc-noteref"><a href="#fn:BLS01" class="footnote" rel="footnote">4</a></sup></li>
<li>identity-based encryption<sup id="fnref:BF01:1" role="doc-noteref"><a href="#fn:BF01" class="footnote" rel="footnote">5</a></sup></li>
<li>additively-homomorphic encryption with support for one multiplication<sup id="fnref:BGN05" role="doc-noteref"><a href="#fn:BGN05" class="footnote" rel="footnote">19</a></sup></li>
<li>succinct zero-knowledge proofs<sup id="fnref:GGPR12e:1" role="doc-noteref"><a href="#fn:GGPR12e" class="footnote" rel="footnote">1</a></sup></li>
</ul>
<p class="info">An interesting pattern to notice here is how pairings evolved from a <em>cryptanalytic tool</em> used to break cryptosystems, to a <strong>constructive tool</strong> used to build cryptosystems.
Interestingly, the same pattern also arose in the development of lattice-based cryptography.</p>
<h2 id="arithmetic-tricks-with-pairings">Arithmetic tricks with pairings</h2>
<p>There are a few tricks cryptographers often use when dealing with pairings in their proofs of correctness or security of a cryptosystem.</p>
<p>The most obvious trick, <strong>“multiplying in the exponent”</strong>, comes from the bilinearity property.</p>
<p>\begin{align}
e(u^a, v^b) = e(u, v)^{ab}
\end{align}</p>
<p>Bilinearity also implies the following trick:
\begin{align}
e(u, v^b) = e(u, v)^b
\end{align}
Or, alternatively:
\begin{align}
e(u^a, v) = e(u, v)^a
\end{align}</p>
<p>Another trick, which is just an analogous way of defining bilinearity, is:
\begin{align}
e(u, v\cdot w) = e(u, v)\cdot e(u, w)
\end{align}</p>
<p class="info"><strong>Why does this work?</strong> Let $y,z$ denote the discrete logs (w.r.t. $g_2$) of $v$ and $w$, respectively.
Then, we have:
\begin{align}
e(u, v\cdot w)
&= e(u, g_2^y \cdot g_2^z)\\<br />
&= e(u, g_2^{y + z})\\<br />
&= e(u, g_2)^{y + z}\\<br />
&= e(u, g_2)^y \cdot e(u, g_2)^z\\<br />
&= e(u, g_2^y) \cdot e(u, g_2^z)\\<br />
&= e(u, v)\cdot e(u, w)
\end{align}</p>
<p>Or, alternatively:
\begin{align}
e(u, v / w) = \frac{e(u, v)}{e(u, w)}
\end{align}</p>
<h2 id="applications-of-pairings">Applications of pairings</h2>
<h3 id="tripartite-diffie-hellman">Tripartite Diffie-Hellman</h3>
<p>This protocol was introduced by Joux in 2000<sup id="fnref:Joux00:1" role="doc-noteref"><a href="#fn:Joux00" class="footnote" rel="footnote">18</a></sup> and assumes a <strong>symmetric pairing</strong>: i.e., where \(\Gr_1 = \Gr_2 = \langle g\rangle \stackrel{\mathsf{def}}{=} \Gr\).</p>
<p>We have three parties Alice, Bob and Charles with secret keys $a, b$, and $c$ (respectively).
They send each other their public keys $g^a, g^b, g^c$ and agree on a shared secret key $k = e(g, g)^{abc}$.<sup id="fnref:dhe" role="doc-noteref"><a href="#fn:dhe" class="footnote" rel="footnote">20</a></sup></p>
<p>How?</p>
<p>Consider Alice’s point of view.
She gets $g^b$ and $g^c$ from Bob and Charles.
First, she can use her secret $a$ to compute $g^{ab}$.
Second, she can use the pairing to compute $e(g^{ab}, g^c) = e(g, g)^{abc} = k$.</p>
<p>By symmetry, all other players can do the same and agree on the same $k$.</p>
<p class="info">The protocol can be generalized to <a href="#asymmetric-pairings"><strong>a</strong>symmetric pairings</a> too, where $\Gr_1 \neq \Gr_2$.</p>
<h3 id="bls-signatures">BLS signatures</h3>
<p>Boneh, Lynn and Shacham give a very short signature scheme from pairings<sup id="fnref:BLS01:2" role="doc-noteref"><a href="#fn:BLS01" class="footnote" rel="footnote">4</a></sup>, which works as follows:</p>
<ul>
<li>Assume $\Gr_2 = \langle g_2 \rangle$ and that there exists a hash function $H : \{0,1\}^* \rightarrow \Gr_1$ modeled as a random oracle.</li>
<li>The secret key is $s \in \Zp$ while the public key is $\pk = g_2^s \in \Gr_2$.</li>
<li>To sign a message $m$, the signer computes $\sigma = H(m)^s\in \Gr_1$.</li>
<li>To verify a signature $\sigma$ on $m$ under public key $\pk$, one checks if $e(\sigma, g_2) \stackrel{?}{=} e(H(m), \pk)$</li>
</ul>
<p>Notice that correctly-computed signatures will always verify since:
\begin{align}
e(\sigma, g_2) \stackrel{?}{=} e(H(m), \pk) \Leftrightarrow\\<br />
e(H(m)^s, g_2) \stackrel{?}{=} e(H(m), g_2^s) \Leftrightarrow\\<br />
e(H(m), g_2)^s \stackrel{?}{=} e(H(m), g_2)^s \Leftrightarrow\\<br />
e(H(m), g_2) = e(H(m), g_2)
\end{align}</p>
<p>See the BLS paper<sup id="fnref:BLS01:3" role="doc-noteref"><a href="#fn:BLS01" class="footnote" rel="footnote">4</a></sup> for how to prove that no attacker can forge BLS signatures given access to $\pk$ and a signing oracle.</p>
<h4 id="cool-properties-of-bls-signatures">Cool properties of BLS signatures</h4>
<p>BLS signatures are quite amazing:</p>
<ol>
<li>They are one of the <strong>simplest</strong> schemes to implement, given access to an elliptic-curve library.</li>
<li>One can <strong>aggregate</strong> many signatures from different public keys on the same message $m$ into a single <em>multi-signature</em> that continues to verify using just 2 pairings.</li>
<li>One can even <strong>aggregate</strong> such signatures across different messages into a single <em>aggregate signature</em>.
<ul>
<li>However, such aggregate signatures take $n+1$ pairings to verify.</li>
</ul>
</li>
<li>One can easily and efficiently<sup id="fnref:TCZplus20" role="doc-noteref"><a href="#fn:TCZplus20" class="footnote" rel="footnote">21</a></sup> build a <strong>threshold</strong> BLS signature scheme, where any subset of $\ge t$ out of $n$ signers can collaborate to sign a message $m$ but no less than $t$ can ever produce a valid signature.
<ul>
<li>Even better, BLS threshold signatures are <strong>deterministic</strong> and give rise to <em>threshold verifiable random functions (VRFs)</em> which are useful for generating randomness on chain.</li>
</ul>
</li>
<li>One can define very-efficient <strong>blind</strong>-variants of BLS signatures, where the signer can sign a message $m$ without learning the message $m$.</li>
<li>BLS signatures are very <strong>efficient</strong> in practice.
<ul>
<li>As far as I remember, they are the most efficient scheme for (1) multi-signatures, (2) aggregate signatures and (3) threshold signatures</li>
<li>For single-signer BLS, they are slower than Schnorr signatures over non-pairing-friendly curves</li>
</ul>
</li>
</ol>
<p class="warning">If you find yourself confused between the various notions of multi-signatures, aggregate signatures and threshold signatures, see <a href="https://docs.google.com/presentation/d/1G4XGqrBLwqMyDQce_xpPQUEMOK4lFrneuvGYU3MVDsI/edit?usp=sharing">my slides</a>.</p>
<h3 id="identity-based-encryption-ibe">Identity-based encryption (IBE)</h3>
<p>In an IBE scheme, one can encrypt directly to a user-friendly email address (or a phone number), instead of a cumbersome public key which is difficult to remember or type-in correctly.</p>
<p>Boneh and Franklin give a very efficient IBE scheme from pairings<sup id="fnref:BF01:2" role="doc-noteref"><a href="#fn:BF01" class="footnote" rel="footnote">5</a></sup>.</p>
<p>For IBE to work, a trusted third-party (TTP) called a <strong>private key generate (PKG)</strong> must be introduced, who will issue secret keys to users based on their email addresses.
This PKG has a <strong>master secret key (MSK)</strong> $\msk \in \Zp$ with an associated <strong>master public key (MPK)</strong> $\mpk = g_2^s$, where $\langle g_2 \rangle = \Gr_2$.</p>
<p>The $\mpk$ is made public and can be used to encrypt a message to any user given their email address.
Crucially, the PKG must keep the $\msk$ secret.
Otherwise, an attacker who steals it can derive any user’s secret key and decrypt everyone’s messages.</p>
<p class="warning">As you can tell the PKG is a central point of failure: theft of the $\msk$ compromises everyone’s secrecy.
To mitigate against this, the PKG can be decentralized into multiple authorities such that a threshold number of authorities must be compromised in order to steal the $\msk$.</p>
<p>Let $H_1 : \{0,1\}^* \rightarrow \Gr_1^*$ and $H_T : \Gr_T \rightarrow \{0,1\}^n$ be two hash functions modeled as random oracles.
To encrypt an $n$-bit message $m$ to a user with email address $id$, one computes:
\begin{align}
g_{id} &= e(H_1(id), \mpk) \in \Gr_T\\<br />
r &\randget \Zp\\<br />
\label{eq:ibe-ctxt}
c &= \left(g_2^r, m \xor H_T\left(\left(g_{id}\right)^r\right)\right) \in \Gr_2\times \{0,1\}^n
\end{align}</p>
<p>To decrypt, the user with email address $id$ must first obtain their <strong>decryption secret key</strong> $\dsk_{id}$ from the PKG.
For this, we assume the PKG has a way of authenticating the user, before handing them their secret key.
For example this could be done via email.</p>
<p>The PKG computes the user’s decryption secret key as:
\begin{align}
\dsk_{id} = H_1(id)^s \in \Gr_1
\end{align}</p>
<p>Now that the user has their decryption secret key, they can decrypt the ciphertext $c = (u, v)$ from Equation $\ref{eq:ibe-ctxt}$ as:
\begin{align}
m &= v \xor H_T(e(\dsk_{id}, u))
\end{align}</p>
<p>You can see why correctly-encrypted ciphertexts will decrypt successfully, since:
\begin{align}
v \xor H_T(e(\dsk_{id}, u))
&= \left(m \xor H_T\left((g_{id})^r \right)\right) \xor H_T\left(e(H_1(id)^s, g_2^r) \right)\\<br />
&= \left(m \xor H_T\left(e(H_1(id), \mpk )^r \right)\right) \xor H_T\left(e(H_1(id), g_2 )^{rs}\right)\\<br />
&= m \xor \left(H_T\left(e(H_1(id), g_2^s)^r \right) \xor H_T\left(e(H_1(id), g_2 )^{rs}\right)\right)\\<br />
&= m \xor \left(H_T\left(e(H_1(id), g_2 )^{rs}\right) \xor H_T\left(e(H_1(id), g_2 )^{rs}\right)\right)\\<br />
&= m
\end{align}</p>
<p>To see why this scheme is secure under chosen-plaintext attacks, refer to the original paper<sup id="fnref:BF01:3" role="doc-noteref"><a href="#fn:BF01" class="footnote" rel="footnote">5</a></sup>.</p>
<h2 id="how-do-pairings-actually-work">How do pairings actually work?</h2>
<p>Mostly, I have no idea.
How come?
Well, I never really needed to know.
And that’s the beauty of pairings: one can use them in a black-box fashion, with zero awareness of their internals.</p>
<p>Still, let’s take a small peek inside this black box.
Let us consider the popular pairing-friendly <em>BLS12-381</em> curve<sup id="fnref:Edgi22" role="doc-noteref"><a href="#fn:Edgi22" class="footnote" rel="footnote">22</a></sup>, from the family of BLS curves characterized by Barreto, Lynn and Scott<sup id="fnref:BLS02e" role="doc-noteref"><a href="#fn:BLS02e" class="footnote" rel="footnote">23</a></sup>.</p>
<p class="warning"><strong>Public service announcement:</strong>
Some of you might’ve heard about <em>Boneh-Lynn-Shacham (BLS)</em> signatures. Please know that this is a different BLS than the BLS in <em>Barretto-Lynn-Scott</em> curves. Confusingly, both acronyms do share one author, Ben Lynn. (In case this was not confusing enough, wait until you have to work with BLS signatures over BLS12-381 curves.)</p>
<p>For BLS12-381, the three groups $\Gr_1, \Gr_2, \Gr_T$ involved are:</p>
<ul>
<li>The group $\Gr_1$ is a subgroup of an elliptic curve $E(\F_q) = \left\{(x, y) \in (\F_q)^2\ \vert\ y^2 = x^3 + 4 \right\}$ where $\vert\Gr_1\vert = p$</li>
<li>The group $\Gr_2$ is a subgroup of a different elliptic curve $E’(\F_{q^2}) = \left\{(x, y) \in (\F_{q^2})^2\ \vert\ y^2 = x^3 + 4(1+i) \right\}$ where $i$ is the square root of $-1$ and $\vert\Gr_2\vert = p$.</li>
<li>The group $\Gr_T$ are all the $p$th roots of unity in $\F_{q^{k}}$, where $k=12$ is called the <em>embedding degree</em></li>
</ul>
<p>How does the pairing map across these three groups work? Well, the pairing $e(\cdot,\cdot)$ expands to something like:
\begin{align}
\label{eq:pairing-def}
e(u, v) = f_{p, u}(v)^{(q^k - 1)/p}
\end{align}
It’s useful to know that computing a pairing consists of two steps:</p>
<ol>
<li>Evaluating the base $f_{p, u}(v)$, also known as a <strong>Miller loop</strong>, in honor of <a href="#history">Victor Miller’s work</a></li>
<li>Raising this base to the constant exponent $(q^k - 1)/p$, also known as a <strong>final exponentiation</strong>.
<ul>
<li>This step is a few times more expensive than the first</li>
</ul>
</li>
</ol>
<p>For more on the internals, see other resources<sup id="fnref:Cost12" role="doc-noteref"><a href="#fn:Cost12" class="footnote" rel="footnote">24</a></sup>$^,$<sup id="fnref:GPS08" role="doc-noteref"><a href="#fn:GPS08" class="footnote" rel="footnote">25</a></sup>$^,$<sup id="fnref:Mene05" role="doc-noteref"><a href="#fn:Mene05" class="footnote" rel="footnote">26</a></sup>.</p>
<h2 id="implementing-pairing-based-crypto">Implementing pairing-based crypto</h2>
<p>This section discusses various implementation-level details that practitioners can leverage to speed up their implementations.</p>
<h3 id="use-asymmetric-pairings">Use asymmetric pairings!</h3>
<p>The pairing over BLS12-381 is <em><strong>a</strong>symmetric</em>: i.e., $\Gr_1 \ne \Gr_2$ are two <strong>different</strong> groups (of the same order $p$). However, there also exist <strong>symmetric</strong> pairings where $\Gr_1 = \Gr_2$ are the same group.</p>
<p>Unfortunately, <em>“such symmetric pairings only exist on supersingular curves, which places a heavy restriction on either or both of the underlying efficiency and security of the protocol”</em><sup id="fnref:BCMplus15e" role="doc-noteref"><a href="#fn:BCMplus15e" class="footnote" rel="footnote">27</a></sup>.
In other words, such supersingular curves are not as efficient at the same security level as the curves used in <strong>a</strong>symmetric pairings.</p>
<p>Therefore, practitioners today, as far as I am aware, exclusively rely on <strong>a</strong>symmetric pairings due to their higher efficiency when the security level is kept the same.</p>
<h3 id="bls12-381-performance">BLS12-381 performance</h3>
<p>I will give a few key performance numbers for the BLS12-381 curve implemented in Filecoin’s (<a href="https://github.com/filecoin-project/blstrs">blstrs</a> Rust wrapper around the popular <a href="https://github.com/supranational/blst">blst</a> library.
These microbenchmarks were run on a 10-core 2021 Apple M1 Max using <code class="language-plaintext highlighter-rouge">cargo bench</code>.</p>
<h4 id="pairing-computation-times">Pairing computation times</h4>
<!--
alinush@MacBook [~/repos/blstrs] (master %) $ cargo +nightly bench -- pairing_
running 4 tests
test bls12_381::bench_pairing_final_exponentiation ... bench: 276,809 ns/iter (+/- 1,911)
test bls12_381::bench_pairing_full ... bench: 484,718 ns/iter (+/- 2,510)
test bls12_381::bench_pairing_g2_preparation ... bench: 62,395 ns/iter (+/- 4,161)
test bls12_381::bench_pairing_miller_loop ... bench: 148,534 ns/iter (+/- 1,203)
-->
<p>As explained in Eq. \ref{eq:pairing-def}, a pairing involves two steps:</p>
<ul>
<li>a Miller loop computation
<ul>
<li>210 microseconds</li>
</ul>
</li>
<li>a final exponentiation
<ul>
<li>276 microseconds</li>
</ul>
</li>
</ul>
<p>Therefore, a pairing takes around 486 microseconds (i.e., the sum of the two).</p>
<h4 id="exponentiation-times">Exponentiation times</h4>
<p class="warning">The $\Gr_T$ microbenchmarks were done by slightly-modifying the <code class="language-plaintext highlighter-rouge">blstrs</code> benchmarking code <a href="https://github.com/filecoin-project/blstrs/blob/e70aff6505fb6f87f9a13e409c080995bd0f244e/benches/bls12_381/ec.rs#L10">here</a>.
(See the HTML comments of this page for those modifications.)</p>
<!--
alinush@MacBook [~/repos/blstrs] (master *%) $ git diff
diff --git a/benches/bls12_381/ec.rs b/benches/bls12_381/ec.rs
index 639bcad..8dcec20 100644
--- a/benches/bls12_381/ec.rs
+++ b/benches/bls12_381/ec.rs
@@ -167,3 +167,34 @@ mod g2 {
});
}
}
+
+mod gt {
+ use rand_core::SeedableRng;
+ use rand_xorshift::XorShiftRng;
+
+ use blstrs::*;
+ use ff::Field;
+ use group::Group;
+
+ #[bench]
+ fn bench_gt_mul_assign(b: &mut ::test::Bencher) {
+ const SAMPLES: usize = 1000;
+
+ let mut rng = XorShiftRng::from_seed([
+ 0x59, 0x62, 0xbe, 0x5d, 0x76, 0x3d, 0x31, 0x8d, 0x17, 0xdb, 0x37, 0x32, 0x54, 0x06,
+ 0xbc, 0xe5,
+ ]);
+
+ let v: Vec<(Gt, Scalar)> = (0..SAMPLES)
+ .map(|_| (Gt::random(&mut rng), Scalar::random(&mut rng)))
+ .collect();
+
+ let mut count = 0;
+ b.iter(|| {
+ let mut tmp = v[count].0;
+ tmp *= v[count].1;
+ count = (count + 1) % SAMPLES;
+ tmp
+ });
+ }
+}
alinush@MacBook [~/repos/blstrs] (master *%) $ cargo +nightly bench -- mul_assign
Compiling blstrs v0.6.1 (/Users/alinush/repos/blstrs)
Finished bench [optimized] target(s) in 0.75s
Running unittests src/lib.rs (target/release/deps/blstrs-349120dc60ef3711)
running 2 tests
test fp::tests::test_fp_mul_assign ... ignored
test scalar::tests::test_scalar_mul_assign ... ignored
test result: ok. 0 passed; 0 failed; 2 ignored; 0 measured; 115 filtered out; finished in 0.00s
Running benches/blstrs_benches.rs (target/release/deps/blstrs_benches-a6732e3e4e5c6a4d)
running 4 tests
test bls12_381::ec::g1::bench_g1_mul_assign ... bench: 72,167 ns/iter (+/- 1,682)
test bls12_381::ec::g2::bench_g2_mul_assign ... bench: 136,184 ns/iter (+/- 1,300)
test bls12_381::ec::gt::bench_gt_mul_assign ... bench: 497,330 ns/iter (+/- 7,802)
test bls12_381::scalar::bench_scalar_mul_assign ... bench: 14 ns/iter (+/- 0)
test result: ok. 0 passed; 0 failed; 0 ignored; 4 measured; 21 filtered out; finished in 5.30s
-->
<ul>
<li>$\Gr_1$ exponentiations are the fastest
<ul>
<li>72 microseconds</li>
</ul>
</li>
<li>$\Gr_2$ exponentiations are around 2$\times$ slower
<ul>
<li>136 microseconds</li>
</ul>
</li>
<li>$\Gr_T$ exponentiations are around 3.5$\times$ slower than in $\Gr_2$
<ul>
<li>500 microseconds</li>
</ul>
</li>
</ul>
<p class="info"><strong>Note:</strong> These benchmarks pick the exponentiated base randomly and do <strong>not</strong> perform any precomputation on it, which would speed up these times by $2\times$-$4\times$.</p>
<h4 id="multi-exponentiations">Multi-exponentiations</h4>
<p>This is a well-known optimization that I’m including for completeness.</p>
<p>Specifically, many libraries allow you to compute a product $\prod_{0 < i < k} \left(g_i\right)^{x_i}$ of $k$ exponentiations much faster than individually computing the $k$ exponentiations and aggregating their product.</p>
<p>For example, <a href="https://github.com/filecoin-project/blstrs">blstrs</a> seems to be incredibly fast in this regard:</p>
<!--
running 4 tests
test bls12_381::bench_g1_multi_exp ... bench: 760,554 ns/iter (+/- 47,355)
test bls12_381::bench_g1_multi_exp_naive ... bench: 18,575,716 ns/iter (+/- 42,688)
test bls12_381::bench_g2_multi_exp ... bench: 1,876,416 ns/iter (+/- 58,743)
test bls12_381::bench_g2_multi_exp_naive ... bench: 35,272,720 ns/iter (+/- 266,279)
-->
<ul>
<li>a size-256 multi-exponentiation in $\Gr_1$
<ul>
<li>takes 760 microseconds in total, or 3 microseconds per exponentiation!</li>
<li>done naively, it would take 18.5 milliseconds in total, which is $24\times$ longer</li>
</ul>
</li>
<li>a size-256 multi-exponentiation in $\Gr_2$
<ul>
<li>takes 1.88 milliseconds in total, or 7.33 microseconds per exponentiation!</li>
<li>done naively, it would take 35.3 milliseconds, which is $18.8\times$ longer</li>
</ul>
</li>
</ul>
<h4 id="group-element-sizes">Group element sizes</h4>
<ul>
<li>$\Gr_1$ group elements are the smallest
<ul>
<li>e.g., 48 bytes for BLS12-381 or 32 bytes for BN254 curves<sup id="fnref:BN06Pair" role="doc-noteref"><a href="#fn:BN06Pair" class="footnote" rel="footnote">28</a></sup></li>
</ul>
</li>
<li>$\Gr_2$ group elements are 2$\times$ larger
<ul>
<li>e.g., 96 bytes on BLS12-381</li>
</ul>
</li>
<li>$\Gr_T$ elements are 12$\times$ larger
<ul>
<li>In general, for a pairing-friendly curve with <em>embedding degree</em> $k$, they are $k$ times larger</li>
</ul>
</li>
</ul>
<h4 id="switching-between-gr_1-and-gr_2">Switching between $\Gr_1$ and $\Gr_2$</h4>
<p>When designing a pairing-based cryptographic protocol, you will want to carefully pick what to use $\Gr_1$ and what to use $\Gr_2$ for.</p>
<p>For example, in BLS signatures, if you want small signatures, then you would compute the signature $\sigma = H(m)^s \in \Gr_1$ and settle for a slightly-larger public key be in $\Gr_2$.
On the other hand, if you wanted to minimize public key size, then you would let it be in $\Gr_1$ while taking slightly longer to compute the signature in $\Gr_2$.</p>
<p class="warning">Other things will also influence how you use $\Gr_1$ and $\Gr_2$, such as the existence of an isomorphism $\phi : \Gr_2 \rightarrow \Gr_1$ or the ability to hash uniformly into these groups.
In fact, the existence of such an isomorphism separates between two types of asymmetric pairings: Type 2 and Type 3 (see <em>Galbraith et al.</em><sup id="fnref:GPS08:1" role="doc-noteref"><a href="#fn:GPS08" class="footnote" rel="footnote">25</a></sup> for more information on the different types of pairings)</p>
<h4 id="comparison-to-non-pairing-friendly-elliptic-curves">Comparison to non-pairing-friendly elliptic curves</h4>
<p>When compared to an elliptic curve that does not admit pairings, pairing-friendly elliptic curves are around two times slower.</p>
<p>For example, the popular prime-order elliptic curve group <a href="https://ristretto.group/">Ristretto255</a> offers:</p>
<!--
ristretto255/basepoint_mul
time: [10.259 µs 10.263 µs 10.267 µs]
ristretto255/point_mul time: [40.163 µs 40.187 µs 40.212 µs]
-->
<ul>
<li>$\approx 2\times$ faster exponentiations of 40 microseconds
<ul>
<li>which can be sped up to 10 microseconds, using precomputation when the exponentiation base is fixed</li>
</ul>
</li>
<li>32 byte group element sizes</li>
</ul>
<h3 id="multi-pairings">Multi-pairings</h3>
<p>If you recall how a pairing actually works (see Eq. $\ref{eq:pairing-def}$), you’ll notice the following optimization:</p>
<p>Whenever, we have to compute the product of $n$ pairings, we can first compute the $n$ Miller loops and do a single final exponentiation instead of $n$.
This drastically reduces the pairing computation time in many applications.
\begin{align}
\prod_i e(u_i, v_i)
&= \prod_i \left(f_{p, u_i}(v_i)^{(q^k - 1)/p}\right)\\<br />
&= \left(\prod_i f_{p, u_i}(v_i)\right)^{(q^k - 1)/p}
\end{align}</p>
<h2 id="conclusion">Conclusion</h2>
<p>This blog post was supposed to be just a short summary of the <a href="#definition-of-a-pairing">three properties of pairings</a>: bilinearity, non-degeneracy and efficiency.</p>
<p>Unfortunately, I felt compelled to discuss their <a href="#history">fascinating history</a>.
And I couldn’t let you walk away without seeing a few powerful <a href="#applications-of-pairings">cryptographic applications of pairings</a>.</p>
<p>After that, I realized practitioners who implement pairing-based cryptosystems might benefit from knowing a little about their <a href="#how-do-pairings-actually-work">internal workings</a>, since some of these details can be leveraged to speed up <a href="#implementation-details">implementations</a>.</p>
<h2 id="acknowledgements">Acknowledgements</h2>
<p>I would like to thank Dan Boneh for helping me clarify and contextualize the history around Weil, as well as for <a href="https://www.youtube.com/watch?v=1RwkqZ6JNeo">his 2015 Simons talk</a>, which inspired me to do a little more research and write this historical account.</p>
<p>Big thanks to:</p>
<ul>
<li><a href="https://twitter.com/cronokirby">Lúcás Meier</a>, <a href="https://twitter.com/zkproofs">Pratyush Mishra</a>, <a href="https://twitter.com/rel_zeta_tech">Ariel Gabizon</a> and <a href="https://twitter.com/dariofiore0">Dario Fiore</a> for their enlightening points on what “succinct” (S) stands for in <strong>S</strong>NARKs<sup id="fnref:GW10:1" role="doc-noteref"><a href="#fn:GW10" class="footnote" rel="footnote">7</a></sup> and for reminding me that SNARKs with $O(1)$ group elements proof size exist from RSA assumptions<sup id="fnref:LM18:1" role="doc-noteref"><a href="#fn:LM18" class="footnote" rel="footnote">10</a></sup>.</li>
<li><a href="https://twitter.com/swasilyev">Sergey Vasilyev</a> for pointing out typos in the BLS12-381 elliptic curve definitions.</li>
<li><a href="https://twitter.com/BlakeMScurr">@BlakeMScurr</a> for pointing out an incorrect reference to Joux’s work<sup id="fnref:Joux00:2" role="doc-noteref"><a href="#fn:Joux00" class="footnote" rel="footnote">18</a></sup>.</li>
<li><a href="https://twitter.com/conradoplg">Conrado Guovea</a> for pointing me to Victor Miller’s account of how he developed his algorithm for evaluating the Weil pairing (discussed <a href="#first-development-millers-algorithm">here</a>).</li>
<li><a href="https://twitter.com/ChrisPeikert">Chris Peikert</a> for pointing out that there are plenty-fast IBE schemes out there that do not rely on pairings<sup id="fnref:DLP14e" role="doc-noteref"><a href="#fn:DLP14e" class="footnote" rel="footnote">29</a></sup>.</li>
</ul>
<p><strong>PS:</strong> Twitter threads are a pain to search through, so if I missed acknowledging your contribution, please kindly let me know.</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:GGPR12e" role="doc-endnote">
<p><strong>Quadratic Span Programs and Succinct NIZKs without PCPs</strong>, by Rosario Gennaro and Craig Gentry and Bryan Parno and Mariana Raykova, <em>in Cryptology ePrint Archive, Paper 2012/215</em>, 2012, <a href="https://eprint.iacr.org/2012/215">[URL]</a> <a href="#fnref:GGPR12e" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:GGPR12e:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:PGHR13e" role="doc-endnote">
<p><strong>Pinocchio: Nearly Practical Verifiable Computation</strong>, by Bryan Parno and Craig Gentry and Jon Howell and Mariana Raykova, <em>in Cryptology ePrint Archive, Paper 2013/279</em>, 2013, <a href="https://eprint.iacr.org/2013/279">[URL]</a> <a href="#fnref:PGHR13e" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Grot16" role="doc-endnote">
<p><strong>On the Size of Pairing-Based Non-interactive Arguments</strong>, by Groth, Jens, <em>in Advances in Cryptology – EUROCRYPT 2016</em>, 2016 <a href="#fnref:Grot16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:BLS01" role="doc-endnote">
<p><strong>Short Signatures from the Weil Pairing</strong>, by Boneh, Dan and Lynn, Ben and Shacham, Hovav, <em>in Advances in Cryptology — ASIACRYPT 2001</em>, 2001 <a href="#fnref:BLS01" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:BLS01:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:BLS01:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:BLS01:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:BF01" role="doc-endnote">
<p><strong>Identity-Based Encryption from the Weil Pairing</strong>, by Boneh, Dan and Franklin, Matt, <em>in Advances in Cryptology — CRYPTO 2001</em>, 2001 <a href="#fnref:BF01" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:BF01:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:BF01:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:BF01:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:KZG10" role="doc-endnote">
<p><strong>Constant-Size Commitments to Polynomials and Their Applications</strong>, by Kate, Aniket and Zaverucha, Gregory M. and Goldberg, Ian, <em>in ASIACRYPT’10</em>, 2010 <a href="#fnref:KZG10" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:GW10" role="doc-endnote">
<p><strong>Separating Succinct Non-Interactive Arguments From All Falsifiable Assumptions</strong>, by Craig Gentry and Daniel Wichs, <em>in Cryptology ePrint Archive, Report 2010/610</em>, 2010, <a href="https://eprint.iacr.org/2010/610">[URL]</a> <a href="#fnref:GW10" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:GW10:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:BCCplus16" role="doc-endnote">
<p><strong>Efficient Zero-Knowledge Arguments for Arithmetic Circuits in the Discrete Log Setting</strong>, by Jonathan Bootle and Andrea Cerulli and Pyrros Chaidos and Jens Groth and Christophe Petit, <em>in Cryptology ePrint Archive, Report 2016/263</em>, 2016, <a href="https://eprint.iacr.org/2016/263">[URL]</a> <a href="#fnref:BCCplus16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Mica98" role="doc-endnote">
<p><strong>Computationally-Sound Proofs</strong>, by Silvio Micali, <em>in Logic Colloquium ‘95: Proceedings of the Annual European Summer Meeting of the Association of Symbolic Logic</em>, 1998, <a href="https://projecteuclid.org/ebooks/lecture-notes-in-logic/Computationally-Sound-Proofs/chapter/Computationally-Sound-Proofs/lnl/1235415908?tab=ChapterArticleLink">[URL]</a> <a href="#fnref:Mica98" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:LM18" role="doc-endnote">
<p><strong>Subvector Commitments with Application to Succinct Arguments</strong>, by Russell W.F. Lai and Giulio Malavolta, <em>in Cryptology ePrint Archive, Report 2018/705</em>, 2018, <a href="https://eprint.iacr.org/2018/705">[URL]</a> <a href="#fnref:LM18" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:LM18:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:Wiki22Weil" role="doc-endnote">
<p><strong>André Weil — Wikipedia, The Free Encyclopedia</strong>, by Wikipedia contributors, 2022, <a href="https://en.wikipedia.org/w/index.php?title=Andr%C3%A9_Weil&oldid=1124211220">[URL]</a> <a href="#fnref:Wiki22Weil" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:danboneh-shimuranote" role="doc-endnote">
<p>Thanks to Dan Boneh, who contrasted Weil’s definition with a different one by Shimura from his classic book on modular forms. While Shimura’s definition makes it much easier to prove all the properties of the pairing, it defines a pairing of order $n$ as a <strong>sum of $n$ points of order $n^2$</strong>. This makes it hopelessly non-computable. Weil’s definition, on the other hand, involves an evaluation of a very concrete function – there are no exponential-sized sums – but requires much more work to prove all its pairing properties. <a href="#fnref:danboneh-shimuranote" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Weil92" role="doc-endnote">
<p><strong>The Apprenticeship of a Mathematician</strong>, by Weil, Andre, 1992, <a href="https://books.google.ro/books?id=73REHmJ9JNUC">[URL]</a> <a href="#fnref:Weil92" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Mill86Short" role="doc-endnote">
<p><strong>Short Programs for functions on Curves</strong>, by Victor S. Miller, 1986, <a href="https://crypto.stanford.edu/miller">[URL]</a> <a href="#fnref:Mill86Short" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:miller-talk" role="doc-endnote">
<p>Miller tells this story himself in <a href="https://www.youtube.com/watch?v=yK5fYfn6HJg&t=2901s">a talk he gave at Microsoft Research</a> on October 10th, 2010. <a href="#fnref:miller-talk" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:alin-where" role="doc-endnote">
<p>I am unable to find any trace of Miller’s published work on this beyond the manuscript Boneh published in<sup id="fnref:Mill86Short:1" role="doc-noteref"><a href="#fn:Mill86Short" class="footnote" rel="footnote">14</a></sup>. Any pointers would be appreciated. <a href="#fnref:alin-where" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:MVO91" role="doc-endnote">
<p><strong>Reducing Elliptic Curve Logarithms to Logarithms in a Finite Field</strong>, by Menezes, Alfred and Vanstone, Scott and Okamoto, Tatsuaki, <em>in ACM STOC</em>, 1991, <a href="http://doi.acm.org/10.1145/103418.103434">[URL]</a> <a href="#fnref:MVO91" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Joux00" role="doc-endnote">
<p><strong>A One Round Protocol for Tripartite Diffie–Hellman</strong>, by Joux, Antoine, <em>in Algorithmic Number Theory</em>, 2000 <a href="#fnref:Joux00" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:Joux00:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:Joux00:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:BGN05" role="doc-endnote">
<p><strong>Evaluating 2-DNF Formulas on Ciphertexts</strong>, by Boneh, Dan and Goh, Eu-Jin and Nissim, Kobbi, <em>in Theory of Cryptography</em>, 2005 <a href="#fnref:BGN05" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:dhe" role="doc-endnote">
<p>Typically, there will be some key-derivation function $\mathsf{KDF}$ used to derive the key as $k = \mathsf{KDF}(e(g,g)^{abc})$. <a href="#fnref:dhe" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:TCZplus20" role="doc-endnote">
<p><strong>Towards Scalable Threshold Cryptosystems</strong>, by Alin Tomescu and Robert Chen and Yiming Zheng and Ittai Abraham and Benny Pinkas and Guy Golan Gueta and Srinivas Devadas, <em>in IEEE S\&P’20</em>, 2020 <a href="#fnref:TCZplus20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Edgi22" role="doc-endnote">
<p><strong>BLS12-381 For The Rest Of Us</strong>, by Ben Edgington, 2022, <a href="https://hackmd.io/@benjaminion/bls12-381">[URL]</a> <a href="#fnref:Edgi22" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:BLS02e" role="doc-endnote">
<p><strong>Constructing Elliptic Curves with Prescribed Embedding Degrees</strong>, by Paulo S. L. M. Barreto and Ben Lynn and Michael Scott, <em>in Cryptology ePrint Archive, Paper 2002/088</em>, 2002, <a href="https://eprint.iacr.org/2002/088">[URL]</a> <a href="#fnref:BLS02e" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Cost12" role="doc-endnote">
<p><strong>Pairings for beginners</strong>, by Craig Costello, 2012, <a href="https://static1.squarespace.com/static/5fdbb09f31d71c1227082339/t/5ff394720493bd28278889c6/1609798774687/PairingsForBeginners.pdf">[URL]</a> <a href="#fnref:Cost12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:GPS08" role="doc-endnote">
<p><strong>Pairings for cryptographers</strong>, by Steven D. Galbraith and Kenneth G. Paterson and Nigel P. Smart, <em>in Discrete Applied Mathematics</em>, 2008, <a href="http://www.sciencedirect.com/science/article/pii/S0166218X08000449">[URL]</a> <a href="#fnref:GPS08" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:GPS08:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:Mene05" role="doc-endnote">
<p><strong>An Introduction to Pairing-Based Cryptography</strong>, by Alfred Menezes, 2005, <a href="https://www.math.uwaterloo.ca/~ajmeneze/publications/pairings.pdf">[URL]</a> <a href="#fnref:Mene05" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:BCMplus15e" role="doc-endnote">
<p><strong>Subgroup security in pairing-based cryptography</strong>, by Paulo S. L. M. Barreto and Craig Costello and Rafael Misoczki and Michael Naehrig and Geovandro C. C. F. Pereira and Gustavo Zanon, <em>in Cryptology ePrint Archive, Paper 2015/247</em>, 2015, <a href="https://eprint.iacr.org/2015/247">[URL]</a> <a href="#fnref:BCMplus15e" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:BN06Pair" role="doc-endnote">
<p><strong>Pairing-Friendly Elliptic Curves of Prime Order</strong>, by Barreto, Paulo S. L. M. and Naehrig, Michael, <em>in Selected Areas in Cryptography</em>, 2006 <a href="#fnref:BN06Pair" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:DLP14e" role="doc-endnote">
<p><strong>Efficient Identity-Based Encryption over NTRU Lattices</strong>, by Léo Ducas and Vadim Lyubashevsky and Thomas Prest, <em>in Cryptology ePrint Archive, Paper 2014/794</em>, 2014, <a href="https://eprint.iacr.org/2014/794">[URL]</a> <a href="#fnref:DLP14e" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Alin Tomescutl;dr: Pairings, or bilinear maps, are a very powerful mathematical tool for cryptography. Pairings gave us our most succinct zero-knowledge proofs[^GGPR12e]$^,$[^PGHR13e]$^,$[^Grot16], our most efficient threshold signatures[^BLS01], our first usable identity-based encryption (IBE)[^BF01] scheme, and many other efficient cryptosystems[^KZG10]. In this post, I’ll teach you a little about the properties of pairings, their cryptographic applications and their fascinating history. In fact, by the end of this post, some of you might want to spend a year or two in jail. Twitter correction: The original tweet announcing this blog post stated that “SNARKs would not be possible without [pairings]”, with the highlighted S meant to emphasize the “succinctness” of such SNARKs. However, thanks to several folks on Twitter, I realized this is not exactly true and depends on what one means by “succinct.” Specifically, “succinct” SNARKs, in the polylogarithmic proof size sense defined by Gentry and Wichs[^GW10], exist from a plethora of assumptions, including discrete log[^BCCplus16] or random oracles[^Mica98]. Furthermore, “succinct” SNARKs, in the sense of $O(1)$ group elements proof size, exist from RSA assumptions too[^LM18]. What pairings do give us, currently, are SNARKs with the smallest, concrete proof sizes (i.e., in # of bytes). $$ \def\idt{\mathbb{1}_{\Gr_T}} \def\msk{\mathsf{msk}} \def\dsk{\mathsf{dsk}} \def\mpk{\mathsf{mpk}} $$ Preliminaries You are familiar with cyclic groups of prime order (e.g., elliptic curves) Let \(\idt\) denote the identity element of the group $\Gr_T$ Let $x \randget S$ denote randomly sampling an element $x$ from a set $S$ Recall that $\langle g \rangle = \Gr$ denotes $g$ being a generator of the group $\Gr$ Definition of a pairing A pairing, also known as a bilinear map, is a function $e : \Gr_1 \times \Gr_2 \rightarrow \Gr_T$ between three groups $\Gr_1, \Gr_2$ and $\Gr_T$ of prime order $p$, with generators $g_1 = \langle \Gr_1 \rangle, g_2 = \langle \Gr_2 \rangle$ and $g_T = \langle \Gr_T \rangle$, respectively. When $\Gr_1 = \Gr_2$, the pairing is called symmetric. Otherwise, it is asymmetric. Most importantly, a pairing has two useful properties for cryptography: bilinearity and non-degeneracy.Hyperproofs: Faster Merkle proof aggregation without SNARKs2022-11-18T00:00:00+00:002022-11-18T00:00:00+00:00https://alinush.github.io//2022/11/18/Hyperproofs-faster-merkle-proof-aggregation-without-snark<p class="info"><strong>tl;dr:</strong> For now, see our Hyperproofs paper<sup id="fnref:SCPplus22" role="doc-noteref"><a href="#fn:SCPplus22" class="footnote" rel="footnote">1</a></sup>.</p>
<!--more-->
<p hidden="">$$
\def\Adv{\mathcal{A}}
\def\Badv{\mathcal{B}}
\def\vect#1{\mathbf{#1}}
$$</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:SCPplus22" role="doc-endnote">
<p><strong>Hyperproofs: Aggregating and Maintaining Proofs in Vector Commitments</strong>, by Shravan Srinivasan and Alexander Chepurnoy and Charalampos Papamanthou and Alin Tomescu and Yupeng Zhang, <em>in 31st USENIX Security Symposium (USENIX Security 22)</em>, 2022, <a href="https://www.usenix.org/conference/usenixsecurity22/presentation/srinivasan">[URL]</a> <a href="#fnref:SCPplus22" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Alin Tomescutl;dr: For now, see our Hyperproofs paper[^SCPplus22].Lagrange interpolation2022-07-28T10:38:00+00:002022-07-28T10:38:00+00:00https://alinush.github.io//2022/07/28/lagrange-interpolation<p>Recall from <a href="/2020/03/16/polynomials-for-crypto.html">our basics discussion</a> that a <strong>polynomial</strong> $\phi$ of <strong>degree</strong> $d$ is a vector of $d+1$ <strong>coefficients</strong>:</p>
<p>\begin{align}
\phi &= [\phi_0, \phi_1, \phi_2, \dots, \phi_d]
\end{align}</p>
<h2 id="how-to-compute-a-polynomials-coefficients-from-a-bunch-of-its-evaluations">How to compute a polynomial’s coefficients from a bunch of its evaluations</h2>
<p>Given $n$ pairs $(x_i, y_i)_{i\in[n]}$, one can compute or <em>interpolate</em> a degree $\le n-1$ polynomial $\phi(X)$ such that:
\(\phi(x_i)=y_i,\forall i\in[n]\)</p>
<p>Specifically, the <em>Lagrange interpolation</em> formula says that:
\begin{align}
\label{eq:lagrange-formula}
\phi(X) &= \sum_{i\in[n]} y_i \cdot \lagr_i(X),\ \text{where}\ \lagr_i(X) = \prod_{j\in[n],j\ne i} \frac{X-x_j}{x_i-x_j}
\end{align}</p>
<p>This formula is intimidating at first, but there’s a very simple intuition behind it.
The key idea is that $\lagr_i(X)$ is defined so that it has two properties:</p>
<ol>
<li>$\lagr_i(x_i) = 1,\forall i\in[n]$</li>
<li>$\lagr_i(x_j) = 0,\forall j \in [n]\setminus\{i\}$</li>
</ol>
<p>You can actually convince yourself that $\lagr_i(X)$ has these properties by plugging in $x_i$ and $x_j$ to see what happens.</p>
<p class="warning"><strong>Important:</strong> The $\lagr_i(X)$ polynomials are dependent on the set of $x_i$’s only (and thus on $n$)! Specifically each $\lagr_i(X)$ has degree $n-1$ and has a root at each $x_j$ when $j\ne i$!
In this sense, a better notation for them would be $\lagr_i^{[x_i, n]}(X)$ or $\lagr_i^{[n]}(X)$ to indicate this dependence.</p>
<h2 id="example-interpolating-a-polynomial-from-three-evaluations">Example: Interpolating a polynomial from three evaluations</h2>
<p>Consider the following example with $n=3$ pairs of points.
Then, by the Lagrange formula, we have:</p>
\[\phi(X) = y_1 \lagr_1(X) + y_2 \lagr_2(X) + y_3 \lagr_3(X)\]
<p>Next, by applying the two key properties of $\lagr_i(X)$ from above, you can easily check that $\phi(x_i) = y_i,\forall i\in[3]$:
\begin{align}
\phi(x_1) &= y_1 \lagr_1(x_1) + y_2 \lagr_2(x_1) + y_3 \lagr_3(x_1) = y_1 \cdot 1 + y_2 \cdot 0 + y_3 \cdot 0 = y_1\\<br />
\phi(x_2) &= y_1 \lagr_1(x_2) + y_2 \lagr_2(x_2) + y_3 \lagr_3(x_2) = y_1 \cdot 0 + y_2 \cdot 1 + y_3 \cdot 0 = y_2\\<br />
\phi(x_3) &= y_1 \lagr_1(x_3) + y_2 \lagr_2(x_3) + y_3 \lagr_3(x_3) = y_1 \cdot 0 + y_2 \cdot 0 + y_3 \cdot 1 = y_3
\end{align}</p>
<p>An <strong>important detail</strong> is that the degree of the interpolated $\phi(X)$ is $\le n-1$ and not necessarily exactly equal to $n-1$.
To see this, consider interpolating the polynomial $\phi(X)$ such that $\phi(i) = i$ for all $i\in [n]$.
In other words, $x_i = y_i = i$.</p>
<p>The inspired reader might notice that the polynomial $\phi(X) = X$ could satisfy our constraints.
But is this what the Lagrange interpolation will return?
After all, the interpolated $\phi(X)$ is a sum of degree $n-1$ polynomials $\lagr_i(X)$, so could it have degree 1?
Well, it turns out, yes, because things cancel out.
To see this, take a simple example, with $n=3$:
\begin{align}
\phi(X) &=\sum_{i\in [3]} i \cdot \lagr_i(X) = \sum_{i\in [3]} i \cdot \prod_{j\in[3]\setminus{i}} \frac{X - j}{i - j}\\<br />
&= 1\cdot \frac{X-2}{1-2}\frac{X-3}{1-3} + 2\cdot \frac{X-1}{2-1}\frac{X-3}{2-3} + 3\cdot\frac{X-1}{3-1}\frac{X-2}{3-2}\\<br />
&= \frac{X-2}{-1}\frac{X-3}{-2} + 2\cdot \frac{X-1}{1}\frac{X-3}{-1} + 3\cdot \frac{X-1}{2}\frac{X-2}{1}\\<br />
&= \frac{1}{2}(X-2)(X-3) - 2(X-1)(X-3) + \frac{3}{2}(X-1)(X-2)\\<br />
&= \frac{1}{2}[(X-2)(X-3) + 3(X-1)(X-2)] - 2(X-1)(X-3)\\<br />
&= \frac{1}{2}[(X-2)(4X-6)] - 2(X-1)(X-3)\\<br />
&= (X-2)(2X-3) - 2(X-1)(X-3)\\<br />
&= (2X^2 - 4X - 3X + 6) - 2(X^2 - 4X +3)\\<br />
&= (2X^2 - 7X + 6) - 2X^2 + 8X - 6\\<br />
&= X
\end{align}</p>
<h2 id="computational-overhead-of-lagrange-interpolation">Computational overhead of Lagrange interpolation</h2>
<p>If done naively, interpolating $\phi(X)$ using the Lagrange formula in Equation \ref{eq:lagrange-formula} will take $O(n^2)$ time.</p>
<p>However, there are known techniques for computing $\phi(X)$ in $O(n\log^2{n})$ time.
We described <strong>part of</strong> these techniques in a <a href="/2020/03/12/scalable-bls-threshold-signatures.html#our-quasilinear-time-bls-threshold-signature-aggregation">previous blog post</a>, but for the full techniques please refer to the <em>“Modern Computer Algebra”</em> book<sup id="fnref:vG13ModernCh10" role="doc-noteref"><a href="#fn:vG13ModernCh10" class="footnote" rel="footnote">1</a></sup>.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:vG13ModernCh10" role="doc-endnote">
<p><strong>Fast polynomial evaluation and interpolation</strong>, by von zur Gathen, Joachim and Gerhard, Jurgen, <em>in Modern Computer Algebra</em>, 2013 <a href="#fnref:vG13ModernCh10" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Alin TomescuRecall from our basics discussion that a polynomial $\phi$ of degree $d$ is a vector of $d+1$ coefficients: \begin{align} \phi &= [\phi_0, \phi_1, \phi_2, \dots, \phi_d] \end{align} How to compute a polynomial’s coefficients from a bunch of its evaluations Given $n$ pairs $(x_i, y_i)_{i\in[n]}$, one can compute or interpolate a degree $\le n-1$ polynomial $\phi(X)$ such that: \(\phi(x_i)=y_i,\forall i\in[n]\) Specifically, the Lagrange interpolation formula says that: \begin{align} \label{eq:lagrange-formula} \phi(X) &= \sum_{i\in[n]} y_i \cdot \lagr_i(X),\ \text{where}\ \lagr_i(X) = \prod_{j\in[n],j\ne i} \frac{X-x_j}{x_i-x_j} \end{align} This formula is intimidating at first, but there’s a very simple intuition behind it. The key idea is that $\lagr_i(X)$ is defined so that it has two properties: $\lagr_i(x_i) = 1,\forall i\in[n]$ $\lagr_i(x_j) = 0,\forall j \in [n]\setminus\{i\}$ You can actually convince yourself that $\lagr_i(X)$ has these properties by plugging in $x_i$ and $x_j$ to see what happens. Important: The $\lagr_i(X)$ polynomials are dependent on the set of $x_i$’s only (and thus on $n$)! Specifically each $\lagr_i(X)$ has degree $n-1$ and has a root at each $x_j$ when $j\ne i$! In this sense, a better notation for them would be $\lagr_i^{[x_i, n]}(X)$ or $\lagr_i^{[n]}(X)$ to indicate this dependence. Example: Interpolating a polynomial from three evaluations Consider the following example with $n=3$ pairs of points. Then, by the Lagrange formula, we have: \[\phi(X) = y_1 \lagr_1(X) + y_2 \lagr_2(X) + y_3 \lagr_3(X)\] Next, by applying the two key properties of $\lagr_i(X)$ from above, you can easily check that $\phi(x_i) = y_i,\forall i\in[3]$: \begin{align} \phi(x_1) &= y_1 \lagr_1(x_1) + y_2 \lagr_2(x_1) + y_3 \lagr_3(x_1) = y_1 \cdot 1 + y_2 \cdot 0 + y_3 \cdot 0 = y_1\\ \phi(x_2) &= y_1 \lagr_1(x_2) + y_2 \lagr_2(x_2) + y_3 \lagr_3(x_2) = y_1 \cdot 0 + y_2 \cdot 1 + y_3 \cdot 0 = y_2\\ \phi(x_3) &= y_1 \lagr_1(x_3) + y_2 \lagr_2(x_3) + y_3 \lagr_3(x_3) = y_1 \cdot 0 + y_2 \cdot 0 + y_3 \cdot 1 = y_3 \end{align} An important detail is that the degree of the interpolated $\phi(X)$ is $\le n-1$ and not necessarily exactly equal to $n-1$. To see this, consider interpolating the polynomial $\phi(X)$ such that $\phi(i) = i$ for all $i\in [n]$. In other words, $x_i = y_i = i$. The inspired reader might notice that the polynomial $\phi(X) = X$ could satisfy our constraints. But is this what the Lagrange interpolation will return? After all, the interpolated $\phi(X)$ is a sum of degree $n-1$ polynomials $\lagr_i(X)$, so could it have degree 1? Well, it turns out, yes, because things cancel out. To see this, take a simple example, with $n=3$: \begin{align} \phi(X) &=\sum_{i\in [3]} i \cdot \lagr_i(X) = \sum_{i\in [3]} i \cdot \prod_{j\in[3]\setminus{i}} \frac{X - j}{i - j}\\ &= 1\cdot \frac{X-2}{1-2}\frac{X-3}{1-3} + 2\cdot \frac{X-1}{2-1}\frac{X-3}{2-3} + 3\cdot\frac{X-1}{3-1}\frac{X-2}{3-2}\\ &= \frac{X-2}{-1}\frac{X-3}{-2} + 2\cdot \frac{X-1}{1}\frac{X-3}{-1} + 3\cdot \frac{X-1}{2}\frac{X-2}{1}\\ &= \frac{1}{2}(X-2)(X-3) - 2(X-1)(X-3) + \frac{3}{2}(X-1)(X-2)\\ &= \frac{1}{2}[(X-2)(X-3) + 3(X-1)(X-2)] - 2(X-1)(X-3)\\ &= \frac{1}{2}[(X-2)(4X-6)] - 2(X-1)(X-3)\\ &= (X-2)(2X-3) - 2(X-1)(X-3)\\ &= (2X^2 - 4X - 3X + 6) - 2(X^2 - 4X +3)\\ &= (2X^2 - 7X + 6) - 2X^2 + 8X - 6\\ &= X \end{align} Computational overhead of Lagrange interpolation If done naively, interpolating $\phi(X)$ using the Lagrange formula in Equation \ref{eq:lagrange-formula} will take $O(n^2)$ time. However, there are known techniques for computing $\phi(X)$ in $O(n\log^2{n})$ time. We described part of these techniques in a previous blog post, but for the full techniques please refer to the “Modern Computer Algebra” book1. Fast polynomial evaluation and interpolation, by von zur Gathen, Joachim and Gerhard, Jurgen, in Modern Computer Algebra, 2013 ↩Feist-Khovratovich technique for computing KZG proofs fast2021-06-17T00:00:00+00:002021-06-17T00:00:00+00:00https://alinush.github.io//2021/06/17/Feist-Khovratovich-technique-for-computing-KZG-proofs-fast<p class="info"><strong>tl;dr:</strong> Given a polynomial $f(X)$ of degree $m$, can we compute all $n$ <a href="/2020/05/06/kzg-polynomial-commitments.html">KZG</a> proofs for $f(\omega^k), k\in[0,n-1]$ in $O(n\log{n})$ time, where $\omega$ is a primitive $n$th root of unity?
Dankrad Feist and Dmitry Khovratovich<sup id="fnref:FK20" role="doc-noteref"><a href="#fn:FK20" class="footnote" rel="footnote">1</a></sup> give a resounding ‘yes!’</p>
<!--more-->
<p hidden="">$$
\def\Adv{\mathcal{A}}
\def\Badv{\mathcal{B}}
\def\vect#1{\mathbf{#1}}
%\definecolor{myBlueColor}{HTML}{268BD2}
%\definecolor{myPinkColor}{HTML}{D33682}
%\definecolor{myGreenColor}{HTML}{859900}
%\def\mygreen#1{\color{myGreenColor}{#1}}
%\def\mygreen#1{\color{green}{#1}}
%\newcommand{\myblue}[1]{\textcolor{myBlueColor}{#1}}
%\newcommand{\mypink}[1]{\textcolor{myPinkColor}{#1}}
$$</p>
<h2 id="preliminaries">Preliminaries</h2>
<p>First, read:</p>
<ul>
<li><a href="/2020/03/16/polynomials-for-crypto.html">Basics of polynomials</a></li>
<li>The <a href="https://en.wikipedia.org/wiki/Discrete_Fourier_transform_(general)">Discrete Fourier Transform (DFT)</a> for $n$th roots of unity</li>
<li><a href="/2020/05/06/kzg-polynomial-commitments.html">KZG polynomial commitments</a></li>
</ul>
<p>Notation:</p>
<ul>
<li>$g$ generates a “bilinear” group $\Gr$ of prime order $p$, endowed with a <em>bilinear map</em> or <em>pairing</em> $e : \Gr\times\Gr\rightarrow\Gr_T$
<ul>
<li>we use multiplicative notation here: i.e., $g^a$ denotes composing $g$ with itself $a$ times</li>
</ul>
</li>
<li>there exists a primitive $n$th root of unity $\omega$, where $n$ is a power of two</li>
<li>we have $q$-SDH public parameters $\left(g, g^\tau,g^{\tau^2},\dots,g^{\tau^q}\right)$, for $q \le n$</li>
<li>we work with polynomials in $\Zp[X]$ of degree $\le n$</li>
</ul>
<h2 id="related-work">Related work</h2>
<p>Other works that give related techniques to compute proofs fast in KZG-like polynomial commitments are:</p>
<ul>
<li><strong>Authenticated multipoint evaluation trees (AMTs)</strong> (see paper<sup id="fnref:TCZplus20" role="doc-noteref"><a href="#fn:TCZplus20" class="footnote" rel="footnote">2</a></sup> and <a href="/2020/03/12/towards-scalable-vss-and-dkg.html">blogpost</a>)</li>
<li><strong>“How to compute all Pointproofs”</strong> (see paper<sup id="fnref:Tome20How" role="doc-noteref"><a href="#fn:Tome20How" class="footnote" rel="footnote">3</a></sup>), much inspired by Feist and Khovratovich’s technique explained in this blogpost</li>
</ul>
<h2 id="refresher-computing-n-different-kzg-proofs">Refresher: Computing $n$ different KZG proofs</h2>
<p>Let $f(X)$ be a polynomial with coefficients $f_i$:</p>
<p>\begin{align*}
f(X) &= f_m X^m + f_{m-1} X^{m-1} + \cdots f_1 X + f_0\\<br />
&= \sum_{i\in[0,m]} f_i X^i
\end{align*}</p>
<p><a href="/2020/05/06/kzg-polynomial-commitments.html">Recall that</a> a KZG <strong>evaluation proof</strong> $\pi_i$ for $f(\omega^i)$ is a KZG commitment to a <strong>quotient polynomial</strong> $Q_i(X) = \frac{f(X) - f(\omega^i)}{X-\omega^i}$:
\begin{align*}
\pi_i = g^{Q_i(\tau)} = g^{\frac{f(\tau) - f(\omega^i)}{\tau-\omega^i}}
\end{align*}</p>
<p>Computing such a proof takes $O(m)$ time!</p>
<p><strong>But what if we want to compute all</strong> $\pi_i, i\in[0, n)$?</p>
<p>If done naively, this would take $O(nm)$ time, which is <strong>too expensive!</strong></p>
<p class="warning"><strong>Lower bound?</strong>
Is this $O(nm)$ time complexity inherent? After all, to compute $\pi_i$ don’t we first need to compute $Q_i(X)$, which takes $O(m)$ time? As you’ll see below, the answer is <strong>“no!”</strong></p>
<p>Fortunately, Feist and Khovratovich observe that the $Q_i$’s are algebraically-related and so are their $\pi_i$ KZG commitments!
As a result, they observe that computing all $\pi_i$’s does <strong>not</strong> require computing all $Q_i$’s.</p>
<p>Below, we explain how their faster, $O(n\log{n})$-time technique works!</p>
<p>An important <strong>caveat</strong> is that their technique relies on the evaluation points being $\omega^0,\dots,\omega^{n-1}$.</p>
<h2 id="quotient-polynomials-and-their-coefficients">Quotient polynomials and their coefficients</h2>
<p>To understand how the $\pi_i$’s relate to one other, let us look at the coefficients of $Q_i(X)$.</p>
<p>We can show that when dividing $f$ (of degree $m$) by $(X-\omega^i)$ we obtain a <em>quotient polynomial</em> with coefficients $t_0, t_1, \dots, t_{m-1}$ such that:
\begin{align}
\label{eq:div-coeffs-1}
t_{m-1} &= f_m\\<br />
%t_{m-2} &= f_{m-1} + \omega^i \cdot f_m\\<br />
%t_{m-3} &= f_{m-2} + \omega^i \cdot t_{m-2}\\<br />
% &= f_{m-2} + \omega^i \cdot (f_{m-1} + \omega^i \cdot f_m)\\<br />
% &= f_{m-2} + \omega^i \cdot f_{m-1} + \omega^{2i} \cdot f_m\\<br />
%t_{m-4} &= f_{m-3} + \omega^i \cdot t_{m-3}\\<br />
% &= f_{m-3} + \omega^i \cdot (f_{m-2} + \omega^i \cdot f_{m-1} + \omega^{2i} \cdot f_m)\\<br />
% &= f_{m-3} + \omega^i \cdot f_{m-2} + \omega^{2i} \cdot f_{m-1} + \omega^{3i} \cdot f_m\\\
% & \vdots\\<br />
\label{eq:div-coeffs-2}
t_{j} &= f_{j+1} + \omega^i \cdot t_{j+1}, \forall j \in [0, m-1)
% & \vdots\\<br />
% t_0 &= f_1 + \omega^i \cdot f_2 + \omega^{2i} f_3 + \dots + \omega^{m-1} f_m
\end{align}
Note that the $t_i$’s are a function of $f_m, f_{m-1},\dots, f_1$, but not of $f_0$!</p>
<details class="info">
<summary>
<b style="color: #f5222d">Proof:</b> One could prove by induction that the coefficients above are correct.
However, it's easiest to take an example and convince yourself, as shown below for $m=4$.
(Click to expand.)
</summary>
<div style="margin-left: .3em; border-left: .15em solid #f5222d; padding-left: .5em;"><p>
Indeed, the quotient obtained when dividing $f(X) = f_3 X^3 + f_2 X^2 + \dots + f_0$ by $X-\omega^i$ exactly matches Equations \ref{eq:div-coeffs-1} and \ref{eq:div-coeffs-2} above:
<!-- WARNING: No support for cline in MathJax, so that's why this is a .png -->
<div align="center">
<a href="/pictures/fk-division-example.png">
<img style="width:95%" src="/pictures/fk-division-example.png" />
</a>
</div>
Specifically, the quotient's coefficients, as expected, are:
\begin{align}
t_2 &= \color{green}{f_3}\\
t_1 &= f_2 + \omega^i t_2\\
&= \color{blue}{f_2 + \omega^i f_3}\\
t_0 &= f_1 + \omega^i t_1 = f_1 + \omega^i \cdot (f_2 + \omega^i f_3)\\
&= \color{pink}{f_1 + \omega^i f_2 + \omega^{2i} f_3}
\end{align}
</p></div>
</details>
<h2 id="the-relationship-between-kzg-proofs">The relationship between KZG proofs</h2>
<p>Next, let us expand Equations \ref{eq:div-coeffs-1} and \ref{eq:div-coeffs-2} above and get a better sense of the relationship between KZG quotient polynomials:
\begin{align}
\color{green}{t_{m-1}} &= \underline{f_m}\\<br />
\color{blue}{t_{m-2}} &= f_{m-1} + \omega^i \cdot \color{green}{t_{m-1}} =\nonumber\\<br />
&= \underline{f_{m-1} + \omega^i \cdot f_m}\\<br />
\color{red}{t_{m-3}} &= f_{m-2} + \omega^i \cdot \color{blue}{t_{m-2}}\nonumber\\<br />
&= f_{m-2} + \omega^i \cdot (f_{m-1} + \omega^i \cdot f_m)\nonumber\\<br />
&= \underline{f_{m-2} + \omega^i \cdot f_{m-1} + \omega^{2i} \cdot f_m}\\<br />
t_{m-4} &= f_{m-3} + \omega^i \cdot \color{red}{t_{m-3}}\nonumber\\<br />
&= f_{m-3} + \omega^i \cdot (f_{m-2} + \omega^i \cdot f_{m-1} + \omega^{2i} \cdot f_m)\nonumber\\<br />
&= \underline{f_{m-3} + \omega^i \cdot f_{m-2} + \omega^{2i} \cdot f_{m-1} + \omega^{3i} \cdot f_m}\\\
&\hspace{.55em}\vdots\nonumber\\<br />
% t_{j} &= f_{j+1} + \omega^i \cdot t_{j+1}, \forall j \in [0, m-1)\\<br />
% & \vdots\\<br />
t_1 &= \underline{f_2 + \omega^i \cdot f_3 + \omega^{2i} \cdot f_4 + \dots + \omega^{(m-2)i} \cdot f_m}\\<br />
t_0 &= \underline{f_1 + \omega^i \cdot f_2 + \omega^{2i} \cdot f_3 + \dots + \omega^{(m-1)i} \cdot f_m}
\end{align}
As you can see above, the quotient polynomial $Q_i(X) = \sum_{j=0}^{m-1} t_j X^j$ obtained when dividing $f(X)$ by $X-\omega^i$ is:
\begin{align}
Q_i(X) &= f_m \cdot X^{m-1} + {}\\<br />
&+ \left(f_{m-1} + \omega^i \cdot f_m\right) \cdot X^{m-2} + {}\nonumber\\<br />
&+ \left(f_{m-2} + \omega^i \cdot f_{m-1} + \omega^{2i} \cdot f_m\right) \cdot X^{m-3} + {}\nonumber\\<br />
&+ \left(f_{m-3} + \omega^i \cdot f_{m-2} + \omega^{2i} \cdot f_{m-1} + \omega^{3i} \cdot f_m\right) \cdot X^{m-4} + {}\nonumber\\\
&+ \dots + {}\nonumber\\\
&+ \left(f_2 + \omega^i \cdot f_3 + \omega^{2i} \cdot f_4 + \dots + \omega^{(m-2)i} \cdot f_m\right) \cdot X + {}\nonumber\\<br />
&+ \left(f_1 + \omega^i \cdot f_2 + \omega^{2i} \cdot f_3 + \dots + \omega^{(m-1)i} \cdot f_m\right)\nonumber
\end{align}
Factoring out the roots of unity, we can rearrange this as follows:
\begin{align}
\label{eq:HX}
Q_i(X) &= \left(f_m X^{m-1} + f_{m-1} X^{m-2} + \dots + f_1\right) (\omega^i)^0 + {}\\<br />
&+ \left(f_m X^{m-2} + f_{m-1} X^{m-3} + \dots + f_2\right) (\omega^i)^1 + {}\nonumber\\<br />
&+ \left(f_m X^{m-3} + f_{m-1} X^{m-4} + \dots + f_3\right) (\omega^i)^2 + {}\nonumber\\<br />
&+ \dots + {}\nonumber\\<br />
&+ \left(f_m X + f_{m-1}\right) (\omega^i)^{m-2} + {}\nonumber\\<br />
&+ \left(f_m \right) (\omega^i)^{m-1}\nonumber
\end{align}
Baptising the polynomials above as $H_j(X)$, we can rewrite as:
\begin{align}
Q_i(X) &\bydef H_1(X) (\omega^i)^0 + {}\\<br />
&+ H_2(X) (\omega^i)^1 + {}\nonumber\\<br />
&+ \dots + {}\nonumber\\<br />
&+ H_m(X) (\omega^i)^{m-1}\nonumber\\<br />
\end{align}
More succinctly, the quotient polynomial is:
\begin{align}
\label{eq:Qi-poly}
Q_i(X) &= \sum_{k=0}^{m-1} H_{j+1}(X) \cdot (\omega^i)^k
\end{align}</p>
<p class="error"><strong>Note:</strong> At this point, it is not helpful to write down a closed form formula for $H_j(X)$, but we’ll return to it later.</p>
<p>Next, let:
\begin{align}
\label{eq:hj}
h_j = g^{H_j(\tau)},\forall j\in[m]
\end{align}
…denote a KZG commitment to $H_j(X)$.
(We are ignoring for now the actual closed-form formula for the $H_j$’s.)</p>
<p>Recall that
<!--\begin{align}-->
\(\pi_i=g^{Q_i(\tau)}\)
<!--\end{align}-->
denotes a KZG proof for $\omega^i$.</p>
<p>Therefore, applying Equation \ref{eq:Qi-poly} to $\pi_i$’s expression, we get:
\begin{align}
\label{eq:pi-dft-like}
\pi_i = \prod_{j=0}^{m-1} \left(h_{j+1}\right)^{(\omega^i)^j}, \forall i\in[0,n)
\end{align}</p>
<p>But a close look at Equation \ref{eq:pi-dft-like} reveals it is actually a <strong>Discrete Fourier Transform (DFT)</strong> on the $h_j$’s!
Specifically, we can rewrite it as:
\begin{align}
\label{eq:pi-dft}
[ \pi_0, \pi_1, \dots, \pi_{n-1} ] = \mathsf{DFT}_{\Gr}(h_1, h_2, \dots, h_m, h_{m+1},\dots, h_n)
\end{align}
Here, the extra $h_{m+1},\dots,h_n$ (if any) are just commitments to the zero polynomials: i.e., they are the identity element in $\Gr$.
(Also, $\mathsf{DFT}_{\Gr}$ is a DFT on group elements via exponentiations, rather than on field elements via multiplication.)</p>
<p class="info"><strong>Time complexity:</strong> Ignoring the time to compute the $h_j$ commitments, which we have not discussed yet, note that the DFT above would only take $O(n\log{n})$ time!</p>
<p>This (almost) summarizes the <strong>Feist-Khovratovich (FK)</strong> technique!</p>
<p><strong>The key idea?</strong> KZG quotient polynomial commitments are actually related, if the evaluation points are roots of unity.
Specifically, these commitments are the output of a single DFT as per Equation \ref{eq:pi-dft}, which can be computed in quasilinear time!</p>
<p>However, <strong>one key challenge remains</strong>, which we address next: computing the $h_j$ commitments.</p>
<h2 id="computing-the-h_j--gh_jtau-commitments">Computing the $h_j = g^{H_j(\tau)}$ commitments</h2>
<p>To see how the $h_j$’s can be computed fast too, let’s rewrite them from Equation \ref{eq:HX}.
\begin{align}
H_1(X) &= f_m X^{m-1} + f_{m-1} X^{m-2} + \dots + f_1\\<br />
H_2(X) &= f_m X^{m-2} + f_{m-1} X^{m-3} + \dots + f_2\\<br />
H_3(X) &= f_m X^{m-3} + f_{m-1} X^{m-4} + \dots + f_3\\<br />
&\vdots\\<br />
H_m(X) &= f_m X + f_{m-1}\\<br />
H_{m-1}(X) &= f_m
\end{align}
<strong>Key observation:</strong> We can express the $H_j(X)$ polynomials as a <a href="/2020/03/19/multiplying-a-vector-by-a-toeplitz-matrix.html">Toeplitz matrix product</a> between a matrix $\mathbf{F}$ (of $f(X)$’s coefficients) and a column vector $V(X)$ (of the indeterminate variable $X$):
\begin{align}
\begin{bmatrix}
H_1(X)\\<br />
H_2(X)\\<br />
H_3(X)\\<br />
\vdots\\<br />
H_m(X)\\<br />
H_{m-1}(X)\\<br />
\end{bmatrix}
&=
\begin{bmatrix}
f_m & f_{m-1} & f_{m-2} & f_{m-3} & \dots & f_2 & f_1\\<br />
0 & f_m & f_{m-1} & f_{m-2} & \dots & f_3 & f_2\\<br />
0 & 0 & f_m & f_{m-1} & \dots & f_4 & f_3\\<br />
\vdots & & & \ddots & & & \vdots\\<br />
0 & 0 & 0 & 0 & \dots & f_m & f_{m-1}\\<br />
0 & 0 & 0 & 0 & \dots & 0 & f_m
\end{bmatrix}
\cdot
\begin{bmatrix}
X^{m-1}\\<br />
X^{m-2}\\<br />
X^{m-3}\\<br />
\vdots\\<br />
X\\<br />
1
\end{bmatrix}
\\<br />
&\bydef
\mathbf{F} \cdot V(X)
\end{align}
Therefore, the commitments $h_j$ to the $H_j(X)$’s can also be expressed as a Toeplitz matrix product, where “multiplication” is replaced with “exponentation” and the column vector $V(X)$ is replaced by $V(\tau)$:
\begin{align}
\begin{bmatrix}
h_1\\<br />
h_2\\<br />
h_3\\<br />
\vdots\\<br />
h_m\\<br />
h_{m-1}\\<br />
\end{bmatrix}
&=
\begin{bmatrix}
f_m & f_{m-1} & f_{m-2} & f_{m-3} & \dots & f_2 & f_1\\<br />
0 & f_m & f_{m-1} & f_{m-2} & \dots & f_3 & f_2\\<br />
0 & 0 & f_m & f_{m-1} & \dots & f_4 & f_3\\<br />
\vdots & & & \ddots & & & \vdots\\<br />
0 & 0 & 0 & 0 & \dots & f_m & f_{m-1}\\<br />
0 & 0 & 0 & 0 & \dots & 0 & f_m
\end{bmatrix}
\cdot
\begin{bmatrix}
\tau^{m-1}\\<br />
\tau^{m-2}\\<br />
\tau^{m-3}\\<br />
\vdots\\<br />
\tau\\<br />
1
\end{bmatrix}
\\<br />
&\bydef
\mathbf{F} \cdot V(\tau)
\end{align}
Fortunately, it is well known that such a matrix product can be computed in $O(m\log{m})$ time (incidentally, also via DFTs).
If you are curious, in a <a href="/2020/03/19/multiplying-a-vector-by-a-toeplitz-matrix.html">previous blogpost</a>, as well as in a short paper<sup id="fnref:Tome20How:1" role="doc-noteref"><a href="#fn:Tome20How" class="footnote" rel="footnote">3</a></sup>, we explain in detail how this works.</p>
<h2 id="conclusion">Conclusion</h2>
<p><strong>We are all done!</strong>
To summarize, to compute all proofs $\pi_i$ for $f(\omega^i)$, the <em>Feist-Khovratovich (FK)</em> technique<sup id="fnref:FK20:1" role="doc-noteref"><a href="#fn:FK20" class="footnote" rel="footnote">1</a></sup> proceeds as follows:</p>
<ol>
<li>Computes all $h_j$’s from Equation \ref{eq:hj} in $O(m\log{m})$ time via a Toeplitz matrix product</li>
<li>Computes all $\pi_i$’s in $O(n\log{n})$ time via a DFT on the $h_j$’s, as per Equation \ref{eq:pi-dft}</li>
</ol>
<p>A few things that we could still talk about, but we are out of time:</p>
<ul>
<li>Implementing this efficiently (see one attempt <a href="https://github.com/alinush/libpolycrypto/blob/fk/libpolycrypto/bench/BenchFk.cpp">here</a>)</li>
<li>Optimizing part of the implementation (see Dankrad’s observation’s in <a href="https://twitter.com/alinush407/status/1360228894851305475">this tweet</a>)</li>
<li>Other techniques for computing proofs on multiple polynomials from the FK paper<sup id="fnref:FK20:2" role="doc-noteref"><a href="#fn:FK20" class="footnote" rel="footnote">1</a></sup></li>
<li>Decreasing KZG verifier time when opening multiple polynomials $(f_i)_{i\in[t]}$ at the same point $x=z^t$, also via the power of DFTs<sup id="fnref:GW21" role="doc-noteref"><a href="#fn:GW21" class="footnote" rel="footnote">4</a></sup></li>
</ul>
<!--
General formula for $H_j$'s:
\begin{align}
H_j(X) \bydef \sum_{k=j}^m f_{k} X^{k-j}, \forall j\in[1,m]
\end{align}
-->
<!--
Sanity check:
H_1(X) = f_1 X^{1-1} + f_2 X^{2-1} + f_3 X^{3-1} + \dots + f_m X^{m-1}
H_2(X) = f_2 X^{2-2} + f_3 X^{3-2} + f_4 X^{4-2} + \dots + f_m X^{m-2}
\vdots
H_j(X) = \prod_{k = j}^m f_k X^{k-j}
\vdots
H_{m-1}(X) = f_{m-1} X^{(m-1) - (m-1)} f_m X^{m-(m-1)} = f_{m-1} + f_m X
H_m(X) = f_m X^{m-m} = f_m
-->
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:FK20" role="doc-endnote">
<p><strong>Fast amortized Kate proofs</strong>, by Dankrad Feist and Dmitry Khovratovich, 2020, <a href="https://github.com/khovratovich/Kate">[URL]</a> <a href="#fnref:FK20" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:FK20:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:FK20:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:TCZplus20" role="doc-endnote">
<p><strong>Towards Scalable Threshold Cryptosystems</strong>, by Alin Tomescu and Robert Chen and Yiming Zheng and Ittai Abraham and Benny Pinkas and Guy Golan Gueta and Srinivas Devadas, <em>in IEEE S\&P’20</em>, 2020 <a href="#fnref:TCZplus20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Tome20How" role="doc-endnote">
<p><strong>How to compute all Pointproofs</strong>, by Alin Tomescu, <em>in Cryptology ePrint Archive, Report 2020/1516</em>, 2020, <a href="https://eprint.iacr.org/2020/1516">[URL]</a> <a href="#fnref:Tome20How" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:Tome20How:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:GW21" role="doc-endnote">
<p><strong>fflonk: a Fast-Fourier inspired verifier efficient version of PlonK</strong>, by Ariel Gabizon and Zachary J. Williamson, <em>in Cryptology ePrint Archive, Report 2021/1167</em>, 2021, <a href="https://ia.cr/2021/1167">[URL]</a> <a href="#fnref:GW21" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Alin Tomescutl;dr: Given a polynomial $f(X)$ of degree $m$, can we compute all $n$ KZG proofs for $f(\omega^k), k\in[0,n-1]$ in $O(n\log{n})$ time, where $\omega$ is a primitive $n$th root of unity? Dankrad Feist and Dmitry Khovratovich[^FK20] give a resounding ‘yes!’Basic number theory2021-04-15T00:00:00+00:002021-04-15T00:00:00+00:00https://alinush.github.io//2021/04/15/basic-number-theory<!-- TODO: totient; gcd's -->
<h2 id="multiplicative-inverses-modulo-m">Multiplicative inverses modulo $m$</h2>
<p>The multiplicative group of integers modulo $m$ is defined as:
\begin{align}
\Z_m^* = \{a\ |\ \gcd(a,m) = 1\}
\end{align}
But why?
This is because Euler’s theorem says that:
\begin{align}
\gcd(a,m) = 1\Rightarrow a^{\phi(m)} = 1
\end{align}
This in turn, implies that every element in $\Z_m^*$ has an inverse, since:
\begin{align}
a\cdot a^{\phi(m) - 1} &= 1
\end{align}
Thus, for a prime $p$, all elements in $\Z_p^* = \{1,2,\dots, p-1\}$ have inverses.
Specifically, the inverse of $a \in \Z_p^*$ is $a^{p-2}$.</p>Alin TomescuMultiplicative inverses modulo $m$ The multiplicative group of integers modulo $m$ is defined as: \begin{align} \Z_m^* = \{a\ |\ \gcd(a,m) = 1\} \end{align} But why? This is because Euler’s theorem says that: \begin{align} \gcd(a,m) = 1\Rightarrow a^{\phi(m)} = 1 \end{align} This in turn, implies that every element in $\Z_m^*$ has an inverse, since: \begin{align} a\cdot a^{\phi(m) - 1} &= 1 \end{align} Thus, for a prime $p$, all elements in $\Z_p^* = \{1,2,\dots, p-1\}$ have inverses. Specifically, the inverse of $a \in \Z_p^*$ is $a^{p-2}$.What is a Merkle tree?2021-02-25T00:00:00+00:002021-02-25T00:00:00+00:00https://alinush.github.io//2021/02/25/what-is-a-merkle-tree<p class="info"><strong>tl;dr:</strong> In this post, we demystify Merkle trees for beginners.
We give simple, illustrated explanations, we detail applications they are used in and reason about their security formally.</p>
<!-- more -->
<p>For more details, see this full post on <a href="https://decentralizedthoughts.github.io/2020-12-22-what-is-a-merkle-tree/">Decentralized Thoughts</a>.</p>Alin Tomescutl;dr: In this post, we demystify Merkle trees for beginners. We give simple, illustrated explanations, we detail applications they are used in and reason about their security formally. For more details, see this full post on Decentralized Thoughts.Authenticated Dictionaries with Cross-Incremental Proof (Dis)aggregation2020-11-26T00:00:00+00:002020-11-26T00:00:00+00:00https://alinush.github.io//2020/11/26/Authenticated-Dictionaries-with-Cross-Incremental-Proof-Disaggregation<p class="info"><strong>tl;dr:</strong> We build an authenticated dictionary (AD) from Catalano Fiore vector commitments that has constant-sized, aggregatable proofs and supports a stronger notion of cross-incremental proof disaggregation.
Our AD could be used for stateless validation in cryptocurrencies with smart contract execution.
In a future post, we will extend this AD with stronger security, non-membership proofs and append-only proofs, which makes it applicable to transparency logging.</p>
<p>This is joint work with my brilliant (ex-)labmates from MIT, <a href="https://twitter.com/SuperAluex">Alex (Yu) Xia</a> and <a href="https://github.com/znewman01">Zack Newman</a>.</p>
<p><strong>Authenticated dictionaries (ADs)</strong> are an important cryptographic primitive which lies at the core of cryptocurrencies such as <em>Ethereum</em> and of transparency logs such as <em>Certificate Transparency (CT)</em>.
Typically, ADs are constructed by Merkleizing a lexicographically-ordered data structure such as a binary search tree, a prefix tree or a skip list.
However, our work takes a different, more algebraic direction, building upon the <a href="/2020/11/24/Catalano-Fiore-Vector-Commitments.html">Catalano-Fiore (CF) vector commitment (VC) scheme</a>.
This has the advantage of giving us <em>constant-sized</em> proofs which are updatable and aggregatable, with a novel notion of <strong>cross-incrementality</strong>.
Importantly, this combination of feautres is not supported by Merkle trees or any other previous VC scheme.</p>
<!--more-->
<p>In a nutshell, in this post, we:</p>
<ul>
<li>Extend CF with a larger index space to accommodate dictionary keys, obtaining an authenticated dictionary,</li>
<li>Extend our AD to support updating proofs and digests after removing keys from the dictionary,</li>
<li>Introduce a novel notion of <strong>cross-incremental proof (dis)aggregation</strong> w.r.t. different ADs</li>
</ul>
<p>In a future post, we will explain how we:</p>
<ul>
<li>Strengthen our AD’s security to handle more adversarial settings such as transparency logs,</li>
<li>Add proofs of non-membership,</li>
<li>Add append-only proofs.</li>
</ul>
<p>Our algebraic approach is not novel in itself and we relate to the previous line of work that explores building ADs from non-Merkle techniques in <strong>our full paper</strong><sup id="fnref:TXN20e" role="doc-noteref"><a href="#fn:TXN20e" class="footnote" rel="footnote">1</a></sup>.
You can also see a quick comparison in our <a href="https://github.com/alinush/authdict-talk/raw/zkstudyclub/talk.pdf">zkStudyClub slides</a>.</p>
<h2 id="preliminaries">Preliminaries</h2>
<p hidden="">$$
\def\Adv{\mathcal{A}}
\def\Badv{\mathcal{B}}
\def\GenGho{\mathsf{GenGroup}_?}
\def\Ghosz{|\Gho|}
\def\Ghoid{1_{\Gho}}
\def\primes{\mathsf{Primes}}
\def\QRn{\mathsf{QR}_N}
\def\multirootexp{\mathsf{MultiRootExp}}
\def\rootfactor{\mathsf{RootFactor}}
\def\vect#1{\mathbf{#1}}
$$</p>
<p>We often use the following notation:</p>
<ul>
<li>$\lambda$ denotes the security parameter of our schemes</li>
<li>$[n] = \{1,2,\dots, n\}$</li>
<li>We denote a vector using a <strong>bolded</strong> variable $\vect{v} = [v_1, \dots, v_n]$</li>
<li>$\Gho$ denotes the hidden-order group our constructions use
<ul>
<li>e.g., \(\Gho = \ZNs =\{a \mathrel\vert \gcd(a,N) = 1\}\)</li>
</ul>
</li>
<li>Let $D$ be a dictionary over a set of keys $K$ that maps each key $k\in K$ to its value $v = D(k)$</li>
<li>We sometimes use $k\in D$ to indicate that key $k$ has some value in the dictionary</li>
<li>We sometimes use $(k,v)\in D$ notation to indicate that key $k$ has value $v$ in the dictionary</li>
<li>We sometimes use $D’ = D\setminus K$ to refer to the new dictionary $D’$ obtained after removing all keys in $K$ (and their values) from the dictionary $D$</li>
</ul>
<p>This post assumes knowledge of:</p>
<ul>
<li>Greatest common divisor (GCD) of two integers $x, y$ denoted by $\gcd(x,y)$</li>
<li>The Extended Euclidean Algorithm (EEA) for computing Bezout coefficients $x,y\in \Z$ such that $ax + by = \gcd(a,b)$</li>
<li><a href="/2020/11/24/RSA-Accumulators.html">RSA accumulators</a>
<ul>
<li>An RSA accumulator for a set \(T = \{b_1, \dots, b_n\}\) of elements where each $b_i$ can be hashed to a <em>prime representative</em> $e_i$ is \(a = g^{\prod_{i \in [n]} e_i}\).</li>
<li>An RSA <em>membership witness</em> for $b_i$ is just \(w_i = a^{1/e_i} = g^{\prod_{j\in[n], j\ne i} e_j}\).</li>
<li>To verify it, just check $w_i^{e_i} = a$.</li>
<li>Recall all RSA membership witnesses can be computed using an algorithm by <em>Sander et al.</em><sup id="fnref:SSY01" role="doc-noteref"><a href="#fn:SSY01" class="footnote" rel="footnote">2</a></sup> baptised as \(\rootfactor\) by <em>Boneh et al.</em><sup id="fnref:BBF18" role="doc-noteref"><a href="#fn:BBF18" class="footnote" rel="footnote">3</a></sup>.</li>
<li>Specifically, \(\rootfactor(g, (e_i)_{i\in[n]}) = (w_i)_{i\in[n]} = (a^{1/e_i})_{i\in[n]} = \left((g^{\prod_{j\in[n]} e_j})^{1/e_i}\right)_{i\in[n]}\)</li>
</ul>
</li>
<li><a href="/2020/11/24/Catalano-Fiore-Vector-Commitments.html">Catalano-Fiore Vector Commitments</a>
<ul>
<li>Let $H$ be a collision-resistant hash function that maps a vector position $i$ to an $\ell+1$ bit prime $e_i$ such that $2^\ell < e_i < 2^{\ell+1}$</li>
<li>The digest of a vector $\vect{v} = [v_1, \dots, v_n]$ is $d(\vect{v}) = (S, \Lambda)$ where:
<ul>
<li>$S = g^{\prod_{i\in[n]} e_i}$ (i.e., an RSA accumulator over all vector indices $i$)</li>
<li>$\Lambda = \prod_{i\in [n]} (S^{1/e_i})^{v_i}$</li>
<li>Note that $\Lambda$ is a multi-exponentiation, where:
<ul>
<li>The bases are RSA witness $S^{1/e_i}$ for $i$,</li>
<li>The exponents are the elements $v_i$!</li>
</ul>
</li>
</ul>
</li>
<li>A proof $\pi_I = (S_I, \Lambda_I)$ for an $I$-subvector $(v_i)_{i\in I}$ is just the digest of $\vect{v}$ without the positions $i\in I$ in it.
<ul>
<li>$S_I = S^{1/\prod_{i\in I} e_i}$ (i.e., an RSA accumulator over all indices except the ones in $I$)</li>
<li>$\Lambda_I = \prod_{i\in[n]\setminus I} (S_I^{1/e_i})^{v_i}$</li>
<li>Again, note that $\Lambda_I$ is a multi-exponentiation, where:
<ul>
<li>The bases are RSA witnesses $S_I^{1/e_i}$ for each $i\in[n]\setminus I$ (but w.r.t. $S_I$)</li>
<li>The exponents are elements $v_i$ for all $i\in[n]\setminus I$</li>
</ul>
</li>
</ul>
</li>
<li>Digests and proofs are <a href="/2020/11/24/Catalano-Fiore-Vector-Commitments.html#updating-digest">updatable</a></li>
<li>Proofs are <a href="/2020/11/24/Catalano-Fiore-Vector-Commitments.html#disaggregating-proofs">incrementally <em>(dis)aggregatable</em></a></li>
</ul>
</li>
</ul>
<h2 id="authenticated-dictionary-ad-schemes">Authenticated dictionary (AD) schemes</h2>
<p>First, forget about <em>authenticated dictionaries</em> and let’s talk about good old plain <em>dictionaries</em>!
Dictionaries are a set of <strong>key-value pairs</strong> such that each <strong>key</strong> is mapped to one <strong>value</strong>.
(We stick to one value per key here, but one could define dictionaries to have multiple values per key too.)
The keys are elements of a <strong>key space</strong> which, for our purposes, is the set of strings of length $2\lambda$ bits.</p>
<p class="info"><strong>Example:</strong> Your phone’s contacts list is a dictionary: it maps each contact’s phone number (i.e., the key) to that contact’s name (i.e., the value).
Similarly, your French-to-English dictionary maps each French word (i.e., the key) to its English counterpart (i.e., the value).</p>
<p>Second, what does it mean to <em>authenticate</em> a dictionary?
The idea is to outsource storage of the dictionary to a <strong>prover</strong> while allowing one or more <strong>verifiers</strong> to correctly <strong>look up</strong> the values of keys in the dictionary.
For this to work, the verifiers must be able to somehow verify the values of keys claimed by the prover.
It should be clear that if the verifiers store nothing, there is nothing they can verify these claims against.
Thus, verifiers must store <em>something</em>.
Since the goal is to outsource storage of the data structure, verifiers will only store a succinct representation of the dictionary called a <strong>digest</strong>.
Importantly, while the dictionary might be very large (e.g., the contacts list of a social butterfly), the digest will actually be constant-sized (e.g., 32 bytes).</p>
<p>Third, how do verifiers look up in authenticated dictionaries?
Verifiers simply ask the prover for a key’s value!
Then, the prover replies with the value together with a <strong>lookup proof</strong> that the verifier checks against the digest!</p>
<p class="info"><strong>Example:</strong> Some of you might be familiar with Merkle prefix trees.
Consider a “sparse” prefix tree that maps each key to a unique leaf.
This is best explained by Laurie and Kasper<sup id="fnref:LK15" role="doc-noteref"><a href="#fn:LK15" class="footnote" rel="footnote">4</a></sup> but, simply put, each key is hashed to a unique path in the tree whose leaf stores that key’s value.
The <em>digest</em> is the Merkle root hash of this Merkle prefix tree.
A <em>lookup proof</em> is the Merkle sibling path to the key’s value in the prefix tree.</p>
<h2 id="our-updatable-ad-for-stateless-validation">Our updatable AD for stateless validation</h2>
<p>We start with a <strong>simple observation</strong>: the CF VC scheme can be repurposed into an authenticated dictionary scheme by treating the vector indices as the dictionary’s keys<sup id="fnref:obs1" role="doc-noteref"><a href="#fn:obs1" class="footnote" rel="footnote">5</a></sup>.
Recall that CF VCs use a collision-resistant hash function $H$ that maps a vector position $i$ to an $(\ell+1)$-bit prime $e_i$ such that $2^\ell < e_i < 2^{\ell+1}$.</p>
<p>We let $e_k = H(k)$ for each key $k$ in the dictionary.
Then, the dictionary’s digest is:
\begin{align}
S &= g^{\prod_{k\in D} e_k}\\<br />
c &= \prod_{(k,v) \in D} (S^{1/e_k})^v
\end{align}
Note that this is just a CF commitment to a “very sparse” vector, with indices in the key space of the dictionary
(The key space is of size $2^{2\lambda}$ since it contains all strings of length $2\lambda$ bits.)
In other words, the dictionary’s key is the vector’s index while the key’s value is the vector element at that index.
Because of this, all the properties of CF VCs carry over to our authenticated dictionary: constant-sized public parameters, incremental proof (dis)aggregation, proof updates and proof precomputation.</p>
<p>Nonetheless, we further enhance this AD by making it more updatable and more (dis)aggregatable.
We call the resulting AD an <strong>updatable authenticated dictionary (UAD)</strong>.</p>
<p class="info">Note that ADs cannot be obtained in this fashion from any VC scheme.
For example, <a href="https://alinush.github.io/2020/05/06/aggregatable-subvector-commitments-for-stateless-cryptocurrencies.html">KZG-based VCs</a> do not support a sparse set of vector indices (but nonetheless other techniques<sup id="fnref:Feis20Multi" role="doc-noteref"><a href="#fn:Feis20Multi" class="footnote" rel="footnote">6</a></sup> can be used there).
However, some schemes like Catalano-Fiore<sup id="fnref:CF13e" role="doc-noteref"><a href="#fn:CF13e" class="footnote" rel="footnote">7</a></sup> and Boneh et al’s VC<sup id="fnref:BBF18:1" role="doc-noteref"><a href="#fn:BBF18" class="footnote" rel="footnote">3</a></sup> do support sparse indices.
Indeed, Boneh et al.<sup id="fnref:BBF18:2" role="doc-noteref"><a href="#fn:BBF18" class="footnote" rel="footnote">3</a></sup> also build an AD on top of their VC scheme, but it is not as (dis)aggregatable as ours.</p>
<h3 id="updating-the-digest-after-removals">Updating the digest after removals</h3>
<p>One new feature we add is updating the digest after a key and its value are <em>removed</em> from the dictionary.
This is very easy to do thanks to the versatility of CF VCs.
First, recall that the proof for $(k,v)$ is just the digest of the dictionary $D$ but without $(k,v)$ in it.
Thus, if we remove $(k,v)$ from $D$, the new digest is just the proof for $(k,v)$!
If we do multiple removals, we can simply <a href="/2020/11/24/Catalano-Fiore-Vector-Commitments.html#aggregating-proofs">aggregate</a> the proofs of all removed keys, which is just the digest of $D$ without those keys in it.
Thus, the new digest after multiple removals is simply this aggregated proof!</p>
<h3 id="updating-proofs-after-removals">Updating proofs after removals</h3>
<p>We also have to add support for updating proofs after a key (and its value) is <em>removed</em> from the dictionary.
Let’s say we want to update an aggregated proof $\pi_K$ for a set of keys $K$ after removing a single key $\hat{k}$ with proof $\pi_{\hat{k}}$.
Recall that $\pi_K$ is the digest of $D\setminus K$.
Since the updated dictionary will be \(D\setminus \{\hat{k}\}\), the updated proof $\pi_K’$ must be the digest of \((D\setminus \{\hat{k}\}) \setminus K\), which is just \(D\setminus (\{\hat{k}\}\cup K)\).</p>
<p>So we must find a way to go from the digest of $D\setminus K$ and of \(D\setminus\{\hat{k}\}\) to the digest of \(D\setminus (\{\hat{k}\}\cup K)\).
Well, the digest of \(D\setminus (\{\hat{k}\}\cup K)\) is nothing but the aggregated proof for $K$ and $\hat{k}$.
Thus, the updated proof for $K$ is simply the aggregation of the old proof for $K$ with the proof for the removed $\hat{k}$.
Naturally, if multiple keys are being removed, then we just aggregate $\pi_K$ with the proofs for each removed key.</p>
<p class="warning">One thing we’ve glanced over was that if \(K = \{\hat{k}\}\), then this proof update doesn’t really work, since we’d be updating the proof for $\hat{k}$ after removing $\hat{k}$ itself.
This doesn’t make sense unless we updated $\hat{k}$’s lookup proof into a non-membership proof, which we have not defined yet, but will do so in a future post.
<br />
<br />
We’ve also glanced over having $\hat{k}\in K$.
But this is not problematic since, in this case, we have \(D\setminus K = D\setminus (\{\hat{k}\}\cup K)\), so the updated proof $\pi_K’ =\pi_K$.</p>
<h3 id="cross-incremental-proof-aggregation">Cross-incremental proof aggregation</h3>
<p>Our paper’s main contribution is <em>cross-incremental proof aggregation</em> for our AD, a technique for <strong>incrementally</strong> aggregating lookup proofs <em>across different dictionaries</em>.
Recall that we can already (incrementally) aggregate two proofs, one for a set of keys $K_1$ and another for $K_2$, into a single proof for the set of keys $K_1\cup K_2$.
For this to work though, these two proofs must be w.r.t. the same dictionary digest $d$.
However, in some applications, we’ll be dealing with proofs $\pi_i$, each for a set of keys $K_i$ but w.r.t. their own digest $d_i$.
This raises the question of whether such proofs can also be <em>cross-aggregated</em>?
Gorbunov et al.<sup id="fnref:GRWZ20e" role="doc-noteref"><a href="#fn:GRWZ20e" class="footnote" rel="footnote">8</a></sup> answer this question positively for vector commitments and our work extends this to authenticated dictionaries.</p>
<p class="info"><strong>Example:</strong> In stateless validation for smart contracts<sup id="fnref:GRWZ20e:1" role="doc-noteref"><a href="#fn:GRWZ20e" class="footnote" rel="footnote">8</a></sup>, the $i$th’s smart contract’s memory is represented as a dictionary with digest $d_i$.
When this $i$th contract is invoked, the transaction will need to include the subset of memory locations $K_i$ that were accessed by the execution together with their proof $\pi_i$.
When multiple transactions are processed, each proof $\pi_i$ will be w.r.t. a different $d_i$.
Importantly, instead of including each $\pi_i$ in the mined block, we would ideally like to <em>cross-aggregate</em> all $\pi_i$’s into a single proof $\pi$.</p>
<h4 id="proof-of-knowledge-of-co-prime-roots">Proof-of-knowledge of co-prime roots</h4>
<p>The key ingredient behind our incremental cross-aggregation is the <strong>proof-of-knowledge of co-prime roots (PoKCR)</strong> protocol by Boneh et al.<sup id="fnref:BBF18:3" role="doc-noteref"><a href="#fn:BBF18" class="footnote" rel="footnote">3</a></sup>
Recall that PoKCR can be used to convince a verifier who has $\alpha_i$’s and $x_i$’s, that the prover <em>knows</em> $w_i$’s such that:</p>
\[\alpha_i = w_i^{x_i},\ \text{for each}\ i\in[n]\]
<p>Importantly, this protocol requires that the $x_i$’s are pairwise co-prime:</p>
\[\gcd(x_i, x_j) = 1,\forall i,j\in[n], i\ne j\]
<p>To prove knowledge of the $w_i$’s, the prover simply gives the verifier:</p>
\[W=\prod_{i\in [n]} w_i\]
<p>To verify knowledge of $w_i$’s, the verifier (who has $\alpha_i$’s and $x_i$’s) computes \(x^* = \prod_{i\in[n]} x_i\) and checks if:</p>
\[W^{x^*} \stackrel{?}{=} \prod_{i\in [n]} \alpha_i^{x^*/x_i}\]
<p>The trick for the verifier is to do this computation efficiently, since the right-hand side (RHS) involves $n$ exponentiations, each of size $O(\ell n)$ bits.
If done naively, this would take $O(\ell n^2)\ \Gho$ operations.
Fortunately, Boneh et al.<sup id="fnref:BBF18:4" role="doc-noteref"><a href="#fn:BBF18" class="footnote" rel="footnote">3</a></sup> give an $O(\ell n\log{n})$ time algorithm to compute this RHS denoted by:</p>
\[\multirootexp((\alpha_i, x_i)_{i\in [n]}) = \prod_{i\in [n]} \alpha_i^{x^*/x_i}\]
<p>We refer you to Figure 1 in our paper<sup id="fnref:TXN20e:1" role="doc-noteref"><a href="#fn:TXN20e" class="footnote" rel="footnote">1</a></sup> for the $\multirootexp$ algorithm, which simply leverages the recursive nature of the problem.
In fact, the algorithm recurses in a manner very similar to <a href="/2020/11/24/RSA-Accumulators.html#precomputing-all-membership-witnesses-fast">$\rootfactor$</a>.</p>
<p>Importantly, Boneh et al. give an <em>extractor</em> that the PoKCR verifier can use to actually recover the $w_i$’s from the $x_i$’s, $\alpha_i$’s and $W$.
This is what makes the protocol a proof of <em>knowledge</em>.
One of our contributions is speeding up the extraction of <em>all</em> $w_i$’s from $O(\ell n^2\log{n})\ \Gho$ operations down to $O(\ell n\log^2{n})$<sup id="fnref:obs2" role="doc-noteref"><a href="#fn:obs2" class="footnote" rel="footnote">9</a></sup>.
For this, we refer you to our full paper<sup id="fnref:TXN20e:2" role="doc-noteref"><a href="#fn:TXN20e" class="footnote" rel="footnote">1</a></sup>.</p>
<h4 id="using-pokcr-for-incrementally-cross-aggregating-lookup-proofs">Using PoKCR for incrementally cross-aggregating lookup proofs</h4>
<p>Suppose we have a lookup proof $\pi_i$ for a set of keys $K_i$ in a dictionary $D_i$ with digest $d_i = (A_i, c_i)$, where $A_i$ is the RSA accumulator over all keys in the dictionary and $c_i$ is the multi-exponentiation of RSA witnesses (i.e., the part of the proof previously denoted using $\Lambda$).
Note we are changing notation slightly for ease of presentation.</p>
<p>The main observation is that we can aggregate several proofs $\pi_i = (W_i, \Lambda_i)$ w.r.t. different digests $d_i$ via PoKCR because $W_i$ and $\Lambda_i$ are actually prime roots of certain group elements.
To see this, recall from the <a href="#preliminaries">preliminaries</a> that:</p>
<p>\begin{align}
W_i &= A_i^{1/e_{K_i}}\\<br />
\Lambda_i &= \left(\prod_{(k,v)\in D_i\setminus K_i} (A_i^{1/e_k})^{v}\right)^{1/e_{K_i}}
\end{align}</p>
<p>Clearly, $W_i$ is an $e_{K_i}$-th root of $A_i$, which the verifier has.
But what about $\Lambda_i$?
Let $v_k$ be the value of each $k\in K_i$ and rewrite $\Lambda_i$ as:
\begin{align}
\Lambda_i &= \left(\prod_{(k,v)\in D_i\setminus K_i} (A_i^{1/e_k})^{v}\right)^{1/e_{K_i}}\\<br />
&= \left(\frac{\prod_{(k,v)\in D_i} (A_i^{1/e_k})^{v}}{\prod_{k\in K_i} (A_i^{1/e_k})^{v_k}}\right)^{1/e_{K_i}}\\<br />
&= \left(c_i / \prod_{k\in K_i} (A_i^{1/e_k})^{v_k}\right)^{1/e_{K_i}}\\<br />
\end{align}
Thus, if we let \(\alpha_i = c_i / \prod_{k\in K_i} (A_i^{1/e_k})^{v_k}\), then $\Lambda_i$ is an $e_{K_i}$-th root of $\alpha_i$.
Note that the verifier can compute $\alpha_i$ from $c_i, W_i$ and $K_i$ (as we describe later).</p>
<p>To summarize, we have $m$ proofs $\pi_i = (W_i, \Lambda_i)$ each w.r.t. its own $d_i = (A_i, c_i)$ such that, for all $i\in [m]$:
\begin{align}
W_i^{e_{K_i}} &= A_i\\<br />
\Lambda_i^{e_{K_i}} &= \alpha_i
\end{align}</p>
<p>We are almost ready to aggregate with PoKCR, but we cannot yet.
This is because the $e_{K_i}$’s must be pairwise co-prime for PoKCR to work!
However, this is not necessarily the case, since we could have a key $k$ that is both in $K_i$ and in $K_j$ which means $e_{K_i}$ and $e_{K_j}$ will have a common factor $e_k = H(k)$.</p>
<p>Fortunately, we can quickly work around this by using a different hash function $H_i$ for each dictionary $D_i$.
This way, the prime representatives for $k\in K_i$ are computed as $e_k = H_i(k)$, while the prime representatives for $k\in K_j$ are computed as $e_k = H_j(k)$.
As long as one cannot find any pair $(k,k’)$ with $H_i(k) = H_j(k’)$, all the $e_{K_i}$’s will be pairwise co-prime.
This means we can aggregate all $m$ proofs as:
\begin{align}
W &= \prod_{i\in[m]} W_i\\<br />
\Lambda &= \prod_{i\in [m]} \Lambda_i
\end{align}</p>
<p>Importantly, we can do this aggregation <em>incrementally</em>: whenever a new proof arrives, we simply multiply it in the previously cross-aggregated proof.</p>
<h4 id="verifying-cross-aggregated-lookup-proofs">Verifying cross-aggregated lookup proofs</h4>
<p>Suppose a verifier gets a cross-aggregated proof $\pi = (W,\Lambda)$ for a bunch of $K_i$’s w.r.t. their own $d_i = (A_i, c_i),\forall i\in[m]$.
How can he verify $\pi$?
First, the verifier checks the PoKCR that, for each $i\in[m]$, there exists $W_i$ such that $A_i = W_i^{e_{K_i}}$:</p>
\[W^{e^*} \stackrel{?}{=} \multirootexp((A_i, e_{K_i})_{i\in [m]}) = \prod_{i\in [m]} A_i^{e^*/e_{K_i}}\]
<p>Here, $e^*=\prod_{i\in[m]} e_{K_i}$ and each $e_{K_i} = \prod_{k\in K_i} H_i(k)$.
Importantly, the verifier can recover the $W_i$’s using the PoKCR extractor (see Section 3.1 in our full paper<sup id="fnref:TXN20e:3" role="doc-noteref"><a href="#fn:TXN20e" class="footnote" rel="footnote">1</a></sup>).</p>
<p>Second, the verifier checks the PoKCR for each $\alpha_i = \Lambda_i^{e_{K_i}}$.
For this, the verifier must first compute each $\alpha_i = c_i / \prod_{k\in K_i} (A_i^{1/e_k})^{v_k}$, where $v_k$ is the value of each $k \in K_i$ and $e_k = H_i(k)$.
The difficult part is computing all $A_i^{1/e_k}$’s, but this can be done via $\rootfactor(W_i, (e_k)_{k\in K_i})$.
Once the verifier has the $\alpha_i$’s, he can check:</p>
\[\Lambda^{e^*} \stackrel{?}{=} \multirootexp((\alpha_i, e_{K_i})_{i\in [m]}) = \prod_{i\in [m]} \alpha_i^{e^*/e_{K_i}}\]
<p>If both PoKCR checks pass, then the verifier is assured the proof verifies.
Not only that, but the verifier can also disaggregate the cross-aggregated proof as we explain next.</p>
<h4 id="disaggregating-cross-aggregated-proofs">Disaggregating cross-aggregated proofs</h4>
<p>Since the cross-aggregated proof $\pi = (W,\Lambda)$ is a PoKCR proof, this mean the PoKCR extractor can be used to recover the original proofs $(\pi_i)_{i\in[m]}$ that $\pi$ was aggregated from.
How?</p>
<p>Well, we already showed how the verifier must extract the $W_i$’s in the original proofs, which he needs for reconstructing the $\alpha_i$’s to verify the $\Lambda$ part of the cross-aggregated proof.
In a similar fashion, the verifier can also extract all the $\Lambda_i$’s aggregated in $\Lambda$.
This way, the verifier can recover the original proofs.
Note that this implies cross-aggregated proofs are <em>updatable</em> by:</p>
<ol>
<li>Cross-disaggregating them into the original lookup proofs,</li>
<li>Updating these lookup proofs,</li>
<li>And cross-reaggregating them back.</li>
</ol>
<h2 id="conclusion">Conclusion</h2>
<p>To conclude, we show that generalizing CF to a larger key-space results in a versatile <em>authenticated dictionary (AD)</em> scheme that supports updating proofs and digests and supports aggregating proofs across different dictionaries in an incremental fashion.
In a future post, we strengthen the security of this construction, which makes it applicable to more adversarial applications such as transparency logging.
As always, see our full paper for details<sup id="fnref:TXN20e:4" role="doc-noteref"><a href="#fn:TXN20e" class="footnote" rel="footnote">1</a></sup>.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:TXN20e" role="doc-endnote">
<p><strong>Authenticated Dictionaries with Cross-Incremental Proof (Dis)aggregation</strong>, by Alin Tomescu and Yu Xia and Zachary Newman, <em>in Cryptology ePrint Archive, Report 2020/1239</em>, 2020, <a href="https://eprint.iacr.org/2020/1239">[URL]</a> <a href="#fnref:TXN20e" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:TXN20e:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:TXN20e:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:TXN20e:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:TXN20e:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a></p>
</li>
<li id="fn:SSY01" role="doc-endnote">
<p><strong>Blind, Auditable Membership Proofs</strong>, by Sander, Tomas and Ta-Shma, Amnon and Yung, Moti, <em>in Financial Cryptography</em>, 2001 <a href="#fnref:SSY01" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:BBF18" role="doc-endnote">
<p><strong>Batching Techniques for Accumulators with Applications to IOPs and Stateless Blockchains</strong>, by Dan Boneh and Benedikt Bünz and Ben Fisch, <em>in Cryptology ePrint Archive, Report 2018/1188</em>, 2018, <a href="https://eprint.iacr.org/2018/1188">[URL]</a> <a href="#fnref:BBF18" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:BBF18:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:BBF18:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:BBF18:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a> <a href="#fnref:BBF18:4" class="reversefootnote" role="doc-backlink">↩<sup>5</sup></a></p>
</li>
<li id="fn:LK15" role="doc-endnote">
<p><strong>Revocation Transparency</strong>, by Ben Laurie and Emilia Kasper, 2015, <a href="https://www.links.org/files/RevocationTransparency.pdf">[URL]</a> <a href="#fnref:LK15" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:obs1" role="doc-endnote">
<p>We were not the first to make this observation; see the work by Agrawal and Raghuraman<sup id="fnref:AR20" role="doc-noteref"><a href="#fn:AR20" class="footnote" rel="footnote">10</a></sup> <a href="#fnref:obs1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Feis20Multi" role="doc-endnote">
<p><strong>Multi-layer hashmaps for state storage</strong>, by Dankrad Feist, 2020, <a href="https://ethresear.ch/t/multi-layer-hashmaps-for-state-storage/7211/print">[URL]</a> <a href="#fnref:Feis20Multi" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:CF13e" role="doc-endnote">
<p><strong>Vector Commitments and their Applications</strong>, by Dario Catalano and Dario Fiore, <em>in Cryptology ePrint Archive, Report 2011/495</em>, 2011, <a href="https://eprint.iacr.org/2011/495">[URL]</a> <a href="#fnref:CF13e" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:GRWZ20e" role="doc-endnote">
<p><strong>Pointproofs: Aggregating Proofs for Multiple Vector Commitments</strong>, by Sergey Gorbunov and Leonid Reyzin and Hoeteck Wee and Zhenfei Zhang, 2020, <a href="https://eprint.iacr.org/2020/419">[URL]</a> <a href="#fnref:GRWZ20e" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:GRWZ20e:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:obs2" role="doc-endnote">
<p>See Section 3.1 in our full paper<sup id="fnref:TXN20e:5" role="doc-noteref"><a href="#fn:TXN20e" class="footnote" rel="footnote">1</a></sup> <a href="#fnref:obs2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:AR20" role="doc-endnote">
<p><strong>KVaC: Key-Value Commitments for Blockchains and Beyond</strong>, by Shashank Agrawal and Srinivasan Raghuraman, <em>in Cryptology ePrint Archive, Report 2020/1161</em>, 2020, <a href="https://eprint.iacr.org/2020/1161">[URL]</a> <a href="#fnref:AR20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Alin Tomescutl;dr: We build an authenticated dictionary (AD) from Catalano Fiore vector commitments that has constant-sized, aggregatable proofs and supports a stronger notion of cross-incremental proof disaggregation. Our AD could be used for stateless validation in cryptocurrencies with smart contract execution. In a future post, we will extend this AD with stronger security, non-membership proofs and append-only proofs, which makes it applicable to transparency logging. This is joint work with my brilliant (ex-)labmates from MIT, Alex (Yu) Xia and Zack Newman. Authenticated dictionaries (ADs) are an important cryptographic primitive which lies at the core of cryptocurrencies such as Ethereum and of transparency logs such as Certificate Transparency (CT). Typically, ADs are constructed by Merkleizing a lexicographically-ordered data structure such as a binary search tree, a prefix tree or a skip list. However, our work takes a different, more algebraic direction, building upon the Catalano-Fiore (CF) vector commitment (VC) scheme. This has the advantage of giving us constant-sized proofs which are updatable and aggregatable, with a novel notion of cross-incrementality. Importantly, this combination of feautres is not supported by Merkle trees or any other previous VC scheme.Catalano-Fiore Vector Commitments2020-11-24T00:00:00+00:002020-11-24T00:00:00+00:00https://alinush.github.io//2020/11/24/Catalano-Fiore-Vector-Commitments<!-- TODO: Write an intro paragraph here -->
<p>A <em>vector commitment (VC)</em> scheme allows a <strong>prover</strong> with access to a vector $\mathbf{v} = [ v_1, \dots, v_n ]$ to convince any <strong>verifier</strong> that position $i$ in $\mathbf{v}$ stores $v_i$ for any index $i\in[n]$.
Importantly, verifers only store a succinct <strong>digest</strong> of the vector (e.g., a 32-byte hash) rather than the full vector $\mathbf{v}$.</p>
<!--more-->
<p hidden="">$$
\def\Adv{\mathcal{A}}
\def\Badv{\mathcal{B}}
\def\GenGho{\mathsf{GenGroup}_?}
\def\Ghosz{|\Gho|}
\def\Ghoid{1_{\Gho}}
\def\multirootexp{\mathsf{MultiRootExp}}
\def\primes{\mathsf{Primes}}
\def\QRn{\mathsf{QR}_N}
\def\rootfactor{\mathsf{RootFactor}}
\def\shamirtrick{\mathsf{ShamirTrick}}
\def\vect#1{\mathbf{#1}}
$$</p>
<p>Below, we review <em>Catalano and Fiore’s</em> elegant VC scheme<sup id="fnref:CF13e" role="doc-noteref"><a href="#fn:CF13e" class="footnote" rel="footnote">1</a></sup> built from hidden-order groups, extended with the enhancements proposed by Lai and Malavolta<sup id="fnref:LM18" role="doc-noteref"><a href="#fn:LM18" class="footnote" rel="footnote">2</a></sup> and by Campanelli et al.<sup id="fnref:CFGplus20e" role="doc-noteref"><a href="#fn:CFGplus20e" class="footnote" rel="footnote">3</a></sup>
We also make one new, small observation: <a href="#disaggregating-proofs">proof disaggregation</a> in this VC is faster than originally thought, which helps precompute all proofs slightly faster.</p>
<h2 id="preliminaries">Preliminaries</h2>
<p>Let $[n] = \{1,2,\dots, n\}$ and $[a,b] = \{a, a+1,\dots, b-1, b\}$.</p>
<p>We assume the reader is familiar with <a href="/2020/11/24/RSA-accumulators.html">RSA accumulators</a>, which Catalano-Fiore (CF) VCs rely heavily upon.
Also, we assume familiarity with the <a href="/2020/11/24/RSA-accumulators.html#precomputing-all-membership-witnesses-fast">$\rootfactor$ algorithm</a> for computing <strong>all</strong> $e_j$th roots of $g^{\prod_{i\in[n]} e_i}, j\in [n]$.</p>
<h3 id="shamirs-trick">Shamir’s trick</h3>
<p>Given $z^{1/a}$ and $z^{1/b}$ where $z\in\Gho$ and $a,b\in\Z$ with $\gcd(a,b) = 1$, one can compute $z^{1/ab} = (z^{1/a})^y (z^{1/b})^x$ where $x,y$ are the Bezout coefficients for $a,b$ such that $ax+by=1$.</p>
<p>This is because:</p>
\[(z^{1/a})^y (z^{1/b})^x = z^\frac{by}{ab} z^\frac{ax}{ab} = z^\frac{ax+by}{ab} = z^\frac{1}{ab}\]
<p>We often use $z^\frac{1}{ab}\leftarrow \shamirtrick(z^{1/a}, z^{1/b}, a, b)$ to denote performing a Shamir’s trick..</p>
<p>Computing $z^{1/(ab)}$ takes \(O(\max(\vert x\vert,\vert y\vert))\ \Gho\) operations since the Bezout coefficients $(a,b)$ are as big as $(x,y)$.
(Some extra time is needed to compute the Bezout coefficients, but we ignore it here.)</p>
<h2 id="public-parameters">Public parameters</h2>
<p>To set up the VC scheme, a hidden-order group $\Gho$ with generator $g$ must be picked such that nobody knows the order of the group.
Importantly, the RSA problem needs to be hard in this group.
<!-- TODO: reference assumptions -->
Typically, $\Gho = \ZNs$ where $N=pq$ is picked via a secure <em>multi-party computation (MPC) ceremony</em> such that nobody knows the factorization of $N$ (and thus nobody knows the order of $\Gho$; i.e., $\vert\Gho\vert = \phi(N)= (p-1)(q-1)$).</p>
<p>Then, a collision-resistant hash function $H : [n] \rightarrow \primes^{\ell+1}$ must be fixe.
$H$ will map each vector index $i\in [n]$ to a prime $e_i = H(i)$ such that $2^\ell < e_i < 2^{\ell+1}$ where $\ell$ is the maximum size in bits of a vector element $v_i$.
We often use $e_K = \prod_{k\in K} e_k$ to denote a product of a subset of such primes, where $K\subseteq[n]$.</p>
<h2 id="digest-or-commitments">Digest (or commitments)</h2>
<p>Instead of commitments, we’ll stick to the <em>digest</em> terminology.</p>
<p>The digest of a vector $\vect{v} = [ v_1, \dots, v_n ]$ consists of two parts.
First, an <a href="/2020/11/24/RSA-accumulators.html">RSA accumulator</a> over all positions in the vector:
\begin{align}
S &= g^{\prod_{i\in[n]} e_i}
\end{align}</p>
<p>Second, a multi-exponentiation of each <a href="/2020/11/24/RSA-accumulators.html#membership-witnesses">RSA membership witness</a> (w.r.t. $S$) for position $i$ to the value $v_i$:
\begin{align}
\Lambda &= \prod_{i\in [n]} (S^{1/e_i})^{v_i}
\end{align}</p>
<p>We often use $d(\vect{v}) = (S, \Lambda)$ to denote the digest of a vector.</p>
<p class="info">Computing the digest requires computing all $S^{1/e_i}, \forall i\in[n]$.
As described <a href="/2020/11/24/RSA-accumulators.html#precomputing-all-membership-witnesses-fast">before</a>, this can be done in $O(n\log{n})$ group exponentiations via \((S^{1/e_i})_{i\in[n]} \leftarrow \rootfactor(g, (e_i)_{i\in[n]})\).
Since each exponentiation is by an $(\ell+1)$-bit prime $e_i$, this takes $O(\ell n\log{n})\ \Gho$ operations.
Then, $\Lambda$ can be computed with an additional $O(\ell n)\ \Gho$ operations.</p>
<p>The beauty of CF VCs is that, once you understand what a digest looks like, everything follows very naturally.</p>
<h2 id="proofs-are-just-digests">Proofs are just digests</h2>
<p>At minimum, in any VC scheme, a verifier can be convinced of the value $v_i$ of any position $i$ in the vector using a <strong>proof</strong> $\pi_i$.
To convince the verifier of multiple values $(\vect{v}_i)_{i\in I}$, this would require individual proofs for each position $i\in I$.
However, in VCs like CF, a single $I$-subvector proof $\pi_I$ can be constructed for all $(v_i)_{i\in I}$.
Importantly, the size of $\pi_I$ is always constant, independent of $\vert I\vert$.</p>
<p>In CF, an $I$-subvector proof $\pi_I$ is just the digest of the vector $\vect{v}$ but <em>“without”</em> positions $I$ in it.
We abuse notation and denote this by \(\vect{v} \setminus I = (v_i)_{i\in[n] \setminus I}\).
Thus, $\pi_I = d(\vect{v}\setminus I)$.</p>
<p>Let $e_I = \prod_{i\in I} e_i$.
Then, the proof $\pi_I$ is just:
\begin{align}
S_I &= S^{1/e_I} = g^{\prod_{i\in[n]\setminus I} e_i}\\<br />
\Lambda_I &= \prod_{i\in [n]\setminus I} (S_I^{1/e_i})^{v_i}
\end{align}</p>
<p>Observe that the accumulator $S_I$ is over all positions $i$ except the ones in $I$.
Also, observe that the membership witnesses in $\Lambda_I$ are w.r.t. to $S_I$ (not $S$) and that $\Lambda_I$ does not contain witnesses for $i\in I$.
This is what it means to commit to a vector $\vect{v}$ “without” positions $I$ in it.</p>
<p class="info">Computing $\pi_I$ takes $O(\ell(n-\vert I \vert)\log(n-\vert I\vert))\ \Gho$ operations, dominated by the cost to compute \((S_I^{1/e_i})_{i\in I}\) via \(\rootfactor(g, (e_i)_{i\in[n]\setminus I})\).</p>
<p>To verify the proof, one simply “adds back” \(\vect{v}_I\) to \(d(\vect{v}\setminus I) = \pi_I\) and checks it obtains the digest $d(\vect{v}) = (S, \Lambda)$ of the vector:
\begin{align}
S &\stackrel{?}{=} S_I^{e_I}\\<br />
\Lambda &\stackrel{?}{=} \Lambda_I^{e_I} \prod_{i\in I} (S^{1/e_i})^{v_i}
\end{align}</p>
<p class="warning">One problem is that all \((S^{1/e_i})_{i\in I}\) must be computed via \(\rootfactor(S_I, (e_i)_{i\in I})\), which would take $O(\ell \vert I\vert \log\vert I\vert)\ \Gho$ operations.
We explain <a href="#updating-digest">next</a> how this can be done in $O(\ell \vert I \vert)\ \Gho$ operations by adding each $v_i, i\in I$ back into $\pi_I$ <em>sequentially</em>!</p>
<h2 id="updating-digest">Updating digest</h2>
<p>It is possible to update the digest $d(\vect{v})=(S,\Lambda)$ after an element $v_i$ changes to $v_i + \delta_i$.
For this, the RSA membership witness $S^{1/e_i}$ is needed as helper information, which we refer to as an <strong>update key</strong>.</p>
<p>Since $S$ does not contain any information about $v_i$ (just about $i$), $S$ stays the same.
However, $\Lambda$ must change into $\Lambda’$ as follows:
\begin{align}
\Lambda’ = \Lambda \cdot (S^{1/e_i})^{\delta_i}
\end{align}</p>
<p class="info">This technique can be generalized to work with many positions $i\in I$ changing by $\delta_i$.
This either requires all the $S^{1/e_i}$ update keys or an <strong>aggregated update key</strong> \(S_I = S^{1/e_I} = S^{\prod_{i\in I} 1/e_i}\) as auxiliary information.
In this last case, all the \((S^{1/e_i})_{i\in I} \leftarrow \rootfactor(S_I, (e_i)_{i\in I})\) can be computed in $O(\ell \vert I \vert \log \vert I \vert)\ \Gho$ operations from the aggregated update key.
Then the update can be performed as \(\Lambda' = \Lambda \cdot \prod_{i\in I} (S^{1/e_i})^{\delta_i}\)</p>
<h3 id="adding-new-positions-to-the-vector">Adding new positions to the vector</h3>
<p>It is also possible to “extend” the vector $\vect{v}$ of size $n$ to size $n+1$ by adding an extra element $v_{n+1}$.
For this, the digest can be updated as:
\begin{align}
S’ &= S^{e_{n+1}}\\<br />
\Lambda’ &= \Lambda^{e_{n+1}} S^{v_{n+1}} = \dots = \prod_{i\in [n+1]} (S’^{1/e_i})^{v_i}
\end{align}</p>
<p class="info">This can be generalized to adding $\Delta$ new positions, and would take $O(\ell \Delta)\ \Gho$ operations by applying each extension sequentially.
In fact, this how one would verify a subvector proof fast: by adding back the elements being verified to the proof (which can be viewed as a digest) and checking the actual digest is obtained!</p>
<h2 id="updating-proofs">Updating proofs</h2>
<p>Since proofs are just digests they are, in principle, updatable in the same fashion, with a few extra details.</p>
<p>Suppose we have a proof $\pi_I$ for a subvector \(\vect{v}_I\):
\begin{align}
S_I &= S^{1/e_I} = S^{1/\prod_{i\in I} e_i} = g^{\prod_{i\in[n]\setminus I} e_i}\\<br />
\Lambda_I &= \prod_{i\in [n]\setminus I} (S_I^{1/e_i})^{v_i}
\end{align}</p>
<h3 id="case-1-iin-i-changed">Case 1: $i\in I$ changed</h3>
<p>First, suppose a position $i\in I$ changes from \(v_i\) to \(v_i + \delta_i\).
Then, the proof $\pi_I$ remains the same!
This is because the proof contains no information about \(v_i\), since the proof is the digest of $\vect{v}$ without any positions $i\in I$.</p>
<h3 id="case-2-jnotin-i-changed">Case 2: $j\notin I$ changed</h3>
<p>Second, suppose a position $j\notin I$ changes from \(v_j\) to \(v_j+\delta_j\).
In this case, we only need to change $\Lambda_I$ into a $\Lambda_I’$ such that:
\begin{align}
\Lambda_I’ = \Lambda_I \cdot (S_I^{1/e_j})^{\delta_j}
\end{align}</p>
<p>The difficult part is computing $S_I^{1/e_j}$.
We could do it from scratch but that would take $O(\ell (n - \vert I \vert))\ \Gho$ operations.
Instead, as with the <a href="#updating-digest">digest update</a>, we assume we are given the $S^{1/e_j}$ update key associated with the changed position $j$.
Fortunately, we know that we can compute $S_I^{1/e_j} = S^{1/(e_I e_j)}$ using <a href="#shamirs-trick">Shamir’s trick</a> on $S_I = S^{1/e_I}$ and $S^{1/e_j}$!
(This takes $O(\ell \vert I \vert)\ \Gho$ operations which is faster when $|I|$ is much smaller than $n$.)</p>
<p class="info">This technique can also be generalized to updating $\pi_I$ after many positions $j\in J$ changed by $\delta_j$.
(Assume without loss of generality that $J\cap I =\varnothing$ or let $J \leftarrow J \setminus I$ otherwise.)
This either requires all $S_I^{1/e_j}$ update keys or the <em>aggregated update key</em> \(S_{I\cup J} = S^{1/(e_I e_J)} = S^{\prod_{k\in I\cup J} 1/e_k}\) as auxiliary information.
In this last case, all the \((S_I^{1/e_j})_{j\in J} \leftarrow \rootfactor(S_{I\cup J}, (e_j)_{j\in J})\) can be computed in $O(\ell \vert J \vert \log \vert J \vert)\ \Gho$ operations from the aggregated update key.
Then, \(\Lambda_I' = \Lambda_I \cdot \prod_{j\in J} (S_I^{1/e_j})^{\delta_j}\).</p>
<h3 id="case-3-adding-new-positions">Case 3: Adding new positions</h3>
<p>Third, suppose a new position $v_{n+1}$ was added to the vector.
Then the proof update proceeds similar to the <a href="#adding-new-positions-to-the-vector">digest update</a> for adding new positions:
\begin{align}
S_I’ &= S_I^{e_{n+1}}\\<br />
\Lambda_I’ &= \Lambda_I^{e_{n+1}} S_I^{v_{n+1}} = \dots = \prod_{i\in [n+1] \setminus I} (S_I’^{1/e_i})^{v_i}
\end{align}</p>
<h2 id="disaggregating-proofs">Disaggregating proofs</h2>
<p>Suppose we want to <strong>disaggregate</strong> a proof $\pi_I=(S_I,\Lambda_I)$ for $\vect{v}_I$ into a proof $\pi_K=(S_K,\Lambda_K)$ for $\vect{v}_K$, where $K\subset I$.</p>
<p>In other words, we want to go from $d(\vect{v}\setminus I)$ to $d(\vect{v}\setminus K)$.
Since $K\subset I$ we can write $K = I\setminus \Delta$ for some set $\Delta$ of indices.
<!-- Thus, slightly abusing notation, $(\vect{v} \setminus K) = (\vect{v} \setminus (I \setminus \Delta)) = (\vect{v} \setminus I \cup \Delta)$ -->
Thus, all we need to do is add back the elements from $\Delta$ to $\pi_I$.</p>
<p>So, for each $i\in \Delta$, we simply add back \((i,v_i)\) to \(\pi_I\) (as explained <a href="#updating-digest">here</a>) and obtain \(\pi_K\).
This takes $O(\ell\vert\Delta\vert)$ $\Gho$ operations.</p>
<h2 id="aggregating-proofs">Aggregating proofs</h2>
<p>To aggregate two subvector proofs for \(\vect{v}_I\) and \(\vect{v}_J\), things are bit more involved.
Let \(\pi_I = (S_I,\Lambda_I)\) and \(\pi_J = (S_J,\Lambda_J)\) denote the two proofs.
Assume that \(I\cap J = \varnothing\) and if not, just set \(J = J\setminus I\) and disaggregate \(\pi_J\) into \(\pi_{J\setminus I}\).
(Can also set \(I=I\setminus J\) and disaggregate \(\pi_I\) instead.)</p>
<p>Denote the aggregated proof by \(\pi_{I\cup J} = (S_{I \cup J}, \Lambda_{I \cup J})\).
We can easily compute the first part of the proof as:</p>
\[S_{I\cup J} = \shamirtrick(S_I, S_J, e_I, e_J) = S^{1/(e_I e_J)}\]
<p>For the second part, focusing on $\Lambda_I$, recall that:
\begin{align}
\Lambda_I &= \prod_{i\in [n]\setminus I} (S_I^{1/e_i})^{v_i}
\end{align}
We will tweak it as \(\Lambda_I^*\) by removing all elements $v_j, j\in J$ from \(\Lambda_I\):
\begin{align}
\Lambda_I^* &= \Lambda_I / \prod_{j\in J} (S_I^{1/e_j})^{v_j}\\<br />
&= \prod_{i\in [n]\setminus (I\cup J)} (S_I^{1/e_i})^{v_i}
\end{align}
But for this, we need to compute \(S_I^{1/e_j},\forall j\in J\) via $\rootfactor$:
\begin{align}
(S_I^{1/e_j})_{j\in J} \leftarrow \rootfactor(S_{I\cup J}, (e_j)_{j\in J})
\end{align}
In a similar, fashion, we can compute
\begin{align}
\Lambda_J^* &= \prod_{i\in [n]\setminus (I\cup J)} (S_J^{1/e_i})^{v_i}
\end{align}
Next, observe that \(\Lambda_I^*\) and \(\Lambda_J^*\) can be rewritten as:
\begin{align}
\Lambda_I^* &= \left(\prod_{i\in [n]\setminus (I\cup J)} (S^{1/e_i})^{v_i}\right)^{1/e_I}\\<br />
\Lambda_J^* &= \left(\prod_{i\in [n]\setminus (I\cup J)} (S^{1/e_i})^{v_i}\right)^{1/e_J}
\end{align}
Thus, if we apply a Shamir trick on them, we can obtain $\Lambda_{I\cup J}$:
\begin{align}
\Lambda_{I\cup J} &= \shamirtrick(\Lambda_I^*, \Lambda_J^*, e_I, e_J)\\<br />
&= \left(\prod_{i\in [n]\setminus (I\cup J)} (S^{1/e_i})^{v_i}\right)^{1/(e_I e_J)}\\<br />
&= \prod_{i\in [n]\setminus (I\cup J)} (S_{I\cup J}^{1/e_i})^{v_i}
\end{align}</p>
<p class="info">Overall, if $b=\max(\vert I\vert, \vert J \vert)$, this requires Shamir tricks of size $b$ and $\rootfactor$’s of size $b$, so it takes $O(\ell b\log{b})\ \Gho$ operations.</p>
<h2 id="precomputing-all-proofs-via-disaggregation">Precomputing all proofs via disaggregation</h2>
<p>Campanelli et al.<sup id="fnref:CFGplus20e:1" role="doc-noteref"><a href="#fn:CFGplus20e" class="footnote" rel="footnote">3</a></sup> explain how to compute all proofs fast via disaggregation.
The idea is to start with a proof for the whole vector $\vect{v}$ and then disaggregate it into a proof for the left half of the vector and the right half.
Then, one repeats recursively on these two halves.
This determines a tree of subvector proofs, where the leaves store the proofs for the individual vector and each internal node stores an aggregation of its children’s proofs (with the root storing the proof for the full vector).</p>
<!--
NOTE: This was the LaTeX for the tree.
Just be sure to add \usepackage{forest} in the preamble in OS X's "LaTeX it!" if you redo this.
\begin{figure*}[t]
{
\normalsize
\begin{center}
\begin{forest}
for tree={
% fit=band,% spaces the tree out a little to avoid collisions
% fit=tight,% spaces the tree out less
% fit=rectangle,
inner sep=4,
}
[{$\pi_{[1,8]},\{v_1,\dots,v_8\}$}
[{$\pi_{[1,4]},\{v_1,\dots,v_4\}$}
[{$\pi_{[1,2]},\{v_1,v_2\}$}
[{$\pi_1,\{v_1\}$}
[, no edge, tier=odd ]
]
[{$\pi_2,\{v_2\}$}
, tier=odd
]
]
[{$\pi_{[3,4]},\{v_3,v_4\}$}
[{$\pi_3,\{v_3\}$}
[, no edge, tier=odd ]
]
[{$\pi_4,\{v_4\}$}
, tier=odd
]
]
]
[{$\pi_{[5,8]},\{v_5,\dots,v_8\}$}
[{$\pi_{[5,6]},\{v_5,v_6\}$}
[{$\pi_5,\{v_5\}$}
[, no edge, tier=odd ]
]
[{$\pi_6,\{v_6\}$}
, tier=odd
]
]
[{$\pi_{[7,8]},\{v_7,v_8\}$}
[{$\pi_7,\{v_7\}$}
[, no edge, tier=odd ]
]
[{$\pi_8,\{v_8\}$}
, tier=odd
]
]
]
]
\end{forest}
\end{center}
}
\end{figure*}
-->
<p>Here’s an example for $n=8$:</p>
<div align="center"><img style="width:95%" src="/pictures/cf-proof-precomp.png" /></div>
<p>Looked at it differently, clearly one can compute this tree via proof aggregation, which we discuss <a href="#aggregating-proofs">next</a>, by starting with the leaves storing individual proofs for each position $i$ and aggregating up the tree.
The key observation is that one can also compute this tree by starting with the root, which stores the proof for the full vector (i.e., the empty digest), <em>disaggregating</em> this proof into two halves, and recursing on these two halves.</p>
<p class="info">This very much resembles the recursion tree used to implement $\rootfactor$ efficiently (see <a href="/2020/11/24/RSA-accumulators.html#precomputing-all-membership-witnesses-fast">here</a>).
Indeed, if you think about it, this tree implicitly computes the $\rootfactor$ tree too, since each CF proof for $v_i$ has an RSA membership witness for $i$ in it.</p>
<p>Note that Campanelli et al.<sup id="fnref:CFGplus20e:2" role="doc-noteref"><a href="#fn:CFGplus20e" class="footnote" rel="footnote">3</a></sup> claim $O(\ell n\log^2{n})\ \Gho$ operations for this algorithm, but the faster disaggregation explained above actually gives $O(\ell n\log{n})$.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:CF13e" role="doc-endnote">
<p><strong>Vector Commitments and their Applications</strong>, by Dario Catalano and Dario Fiore, <em>in Cryptology ePrint Archive, Report 2011/495</em>, 2011, <a href="https://eprint.iacr.org/2011/495">[URL]</a> <a href="#fnref:CF13e" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:LM18" role="doc-endnote">
<p><strong>Subvector Commitments with Application to Succinct Arguments</strong>, by Russell W.F. Lai and Giulio Malavolta, <em>in Cryptology ePrint Archive, Report 2018/705</em>, 2018, <a href="https://eprint.iacr.org/2018/705">[URL]</a> <a href="#fnref:LM18" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:CFGplus20e" role="doc-endnote">
<p><strong>Vector Commitment Techniques and Applications to Verifiable Decentralized Storage</strong>, by Matteo Campanelli and Dario Fiore and Nicola Greco and Dimitris Kolonelos and Luca Nizzardo, 2020, <a href="https://eprint.iacr.org/2020/149">[URL]</a> <a href="#fnref:CFGplus20e" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:CFGplus20e:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:CFGplus20e:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
</ol>
</div>Alin TomescuA vector commitment (VC) scheme allows a prover with access to a vector $\mathbf{v} = [ v_1, \dots, v_n ]$ to convince any verifier that position $i$ in $\mathbf{v}$ stores $v_i$ for any index $i\in[n]$. Importantly, verifers only store a succinct digest of the vector (e.g., a 32-byte hash) rather than the full vector $\mathbf{v}$.RSA Accumulators2020-11-24T00:00:00+00:002020-11-24T00:00:00+00:00https://alinush.github.io//2020/11/24/RSA-accumulators<p>An <strong>RSA accumulator</strong> is an <em>authenticated set</em> built from cryptographic assumptions in hidden-order groups such as $\mathbb{Z}_N^*$.
RSA accumulators enable a <strong>prover</strong>, who stores the full set, to convince any <strong>verifier</strong>, who only stores a succinct <strong>digest</strong> of the set, of various set relations such as (non)membership, subset or disjointness.
For example, the prover can prove membership of elements in the set to verifiers who have the digest.</p>
<!--more-->
<p hidden="">$$
\def\Adv{\mathcal{A}}
\def\Badv{\mathcal{B}}
\def\GenGho{\mathsf{GenGroup}_?}
\def\Ghosz{|\Gho|}
\def\Ghoid{1_{\Gho}}
\def\primes{\mathsf{Primes}}
\def\QRn{\mathsf{QR}_N}
\def\multirootexp{\mathsf{MultiRootExp}}
\def\rootfactor{\mathsf{RootFactor}}
\def\vect#1{\mathbf{#1}}
$$</p>
<p>The <em>digest</em> of the set is often referred to as the <em>accumulator</em> of the set.
<!-- TODO: Link to RSA assumptions post --></p>
<p>RSA accumulators were introduced by <em>Benaloh and de Mare</em><sup id="fnref:Bd93" role="doc-noteref"><a href="#fn:Bd93" class="footnote" rel="footnote">1</a></sup> and later extended by <em>Li et al.</em><sup id="fnref:LLX07" role="doc-noteref"><a href="#fn:LLX07" class="footnote" rel="footnote">2</a></sup> with non-mmebership proofs.
Recently, Boneh et al.<sup id="fnref:BBF18" role="doc-noteref"><a href="#fn:BBF18" class="footnote" rel="footnote">3</a></sup> extended RSA accumulators with many new features.</p>
<p class="warning">This post is not a full treatment of RSA accumulators, but will be extended over time.</p>
<h2 id="digest-or-accumulator-of-a-set">Digest (or accumulator) of a set</h2>
<p>Let $H$ be a collision-resistant hash function that maps its input to prime numbers.
Let $g$ be the generator of a hidden-order group $\Gho$ where the Strong RSA problem is hard.
<!-- TODO: reference assumptions -->
Let $T = \{b_1, b_2, \dots, b_n\}$ and let $e_i = H(b_i)$ be the <em>prime representative</em> of $b_i$.</p>
<p>The accumulator of $T$ is:</p>
\[a = g^{\prod_{i\in[n]} e_i}\]
<p>Note that this can be computed in $O(n)$ exponentiations in $\Gho$.</p>
<h2 id="membership-witnesses">Membership witnesses</h2>
<p>To prove that $b_i$ is in the accumulator, a <em>membership witness</em> can be computed:</p>
\[w_i = g^{\prod_{j\in[n]\setminus\{i\}} e_j} = a^{1/e_i}\]
<p>Note that the witness is simply the accumulator of the set $T \setminus \{b_i\}$ and is just an $e_i$th root of $a$!</p>
<p class="warning">Unfortunately, computing this root cannot be done using an exponentiation by $1/e_i$ because $e_i$ cannot be inverted without knowing the order of the group $\Gho$.
Instead, computing $w_i$ has to be done by exponentiating $g$ by the $n-1$ different $e_i$’s and thus takes $O(n)$ time, which can be slow.
We explain <a href="#precomputing-all-membership-witnesses-fast">below</a> how this can be done in $O(\log{n})$ amortized time per witness.</p>
<p>To verify the witness against the accumulator $a$, one checks if:</p>
\[a \stackrel{?}{=} w_i^{e_i}\]
<p>In other words, one simply “adds back” $b_i$ to the set accumulated in $w_i$ and checks if the result equals $a$.
This takes one exponentiation in $\Gho$.</p>
<p>Membership witnesses can be easily generalized into <em>batch membership witnesses</em>.
For example, if one wants to prove that all $(b_i)_{i\in I}$ are in the accumulator, they can compute:</p>
\[w_I = g^{\prod_{i\in[n]\setminus I} e_i} = a^{1/ \prod_{i\in I} e_i}\]
<p>The verification proceeds analogously but takes $O(\vert I \vert)$ exponentiations:</p>
\[a \stackrel{?}{=} w_i^{\prod_{i\in I}e_i}\]
<h2 id="precomputing-all-membership-witnesses-fast">Precomputing all membership witnesses fast</h2>
<p>Computing all $n$ membership witnesses naively takes $O(n^2)$ exponentiations in $\Gho$, which does not scale well.
Fortunately, <em>Sander et al.</em><sup id="fnref:SSY01" role="doc-noteref"><a href="#fn:SSY01" class="footnote" rel="footnote">4</a></sup> give a divide-and-conquer approach for computing all witnesses which takes $O(n\log{n})$ exponentiations in $\Gho$.</p>
<p>The key observation is that half of the witnesses require computing $g^{\prod_{i\in[1,n/2]} e_i}$ while the other half require computing $g^{\prod_{i\in[n/2+1,n]} e_i}$.
If done naively, these computations would be repeated many times unnecessarily.
But one could compute the witnesses recursively in a tree-like fashion as follows ($n=8$ in this example):</p>
<div align="center"><img style="width:100%" src="/pictures/rsa-membwit-precomp.png" /></div>
<p class="info"><strong>Intuition:</strong>
You can think of each node as (1) a set of elements and (2) its <em>batch membership witness</em> w.r.t. the accumulator $a$.
Then, this algorithm simply splits the set into two halves and disaggregates the witness into witnesses for the two halves.
This repeats until witnesses for individual elements are obtained at the bottom.</p>
<p>In other words, this algorithm computes every $e_i$th root of $a$ (i.e., $g^{e_1 \dots e_{i-1} e_{i+1} \dots e_{n}}$) for all $i\in[n]$.
The time to compute a tree of size $n$ leaves can be defined recursively as $T(n) = 2T(n/2) + O(n)$ because the amount of work done at the root of a size-$n$ subtree is $O(n)$ exponentiations plus the time to recurse on its two children.
This gives $T(n) = O(n\log{n})$ exponentiations.</p>
<p>Boneh et al.<sup id="fnref:BBF18:1" role="doc-noteref"><a href="#fn:BBF18" class="footnote" rel="footnote">3</a></sup> baptised this algorithm as \(\rootfactor\).
Specifically,</p>
\[\rootfactor(g, (e_i)_{i\in[n]}) = (w_i)_{i\in[n]} = (a^{1/e_i})_{i\in[n]} = \left((g^{\prod_{j\in[n]} e_j})^{1/e_i}\right)_{i\in[n]}\]
<!-- **TODO:** Future post: update membership, aggregate membership, non-membership witnesses + update, subset relations, disjointness relations -->
<h2 id="more-to-come">More to come</h2>
<p>There is so much more to be said about RSA accumulators!
Hopefully, we will update this post over time with:</p>
<ul>
<li>Accumulator updates</li>
<li>Membership witness updates</li>
<li>Cross-incremental (dis)aggregation of membership witnesses<sup id="fnref:TXN20e" role="doc-noteref"><a href="#fn:TXN20e" class="footnote" rel="footnote">5</a></sup></li>
<li>Non-membership witnesses
<ul>
<li>Updates</li>
<li>Aggregation</li>
</ul>
</li>
<li>Disjointness witnesses<sup id="fnref:Tome20" role="doc-noteref"><a href="#fn:Tome20" class="footnote" rel="footnote">6</a></sup></li>
<li>Subset witnesses<sup id="fnref:Tome20:1" role="doc-noteref"><a href="#fn:Tome20" class="footnote" rel="footnote">6</a></sup></li>
</ul>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:Bd93" role="doc-endnote">
<p><strong>One-Way Accumulators: A Decentralized Alternative to Digital Signatures</strong>, by Benaloh, Josh and de Mare, Michael, <em>in EUROCRYPT ‘93</em>, 1994 <a href="#fnref:Bd93" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:LLX07" role="doc-endnote">
<p><strong>Universal Accumulators with Efficient Nonmembership Proofs</strong>, by Li, Jiangtao and Li, Ninghui and Xue, Rui, <em>in Applied Cryptography and Network Security</em>, 2007 <a href="#fnref:LLX07" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:BBF18" role="doc-endnote">
<p><strong>Batching Techniques for Accumulators with Applications to IOPs and Stateless Blockchains</strong>, by Dan Boneh and Benedikt Bünz and Ben Fisch, <em>in Cryptology ePrint Archive, Report 2018/1188</em>, 2018, <a href="https://eprint.iacr.org/2018/1188">[URL]</a> <a href="#fnref:BBF18" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:BBF18:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:SSY01" role="doc-endnote">
<p><strong>Blind, Auditable Membership Proofs</strong>, by Sander, Tomas and Ta-Shma, Amnon and Yung, Moti, <em>in Financial Cryptography</em>, 2001 <a href="#fnref:SSY01" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:TXN20e" role="doc-endnote">
<p><strong>Authenticated Dictionaries with Cross-Incremental Proof (Dis)aggregation</strong>, by Alin Tomescu and Yu Xia and Zachary Newman, <em>in Cryptology ePrint Archive, Report 2020/1239</em>, 2020, <a href="https://eprint.iacr.org/2020/1239">[URL]</a> <a href="#fnref:TXN20e" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Tome20" role="doc-endnote">
<p><strong>How to Keep a Secret and Share a Public Key (Using Polynomial Commitments)</strong>, by Tomescu, Alin, 2020, <a href="https://alinush.github.io/papers/phd-thesis-mit2020.pdf">[URL]</a> <a href="#fnref:Tome20" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:Tome20:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
</ol>
</div>Alin TomescuAn RSA accumulator is an authenticated set built from cryptographic assumptions in hidden-order groups such as $\mathbb{Z}_N^*$. RSA accumulators enable a prover, who stores the full set, to convince any verifier, who only stores a succinct digest of the set, of various set relations such as (non)membership, subset or disjointness. For example, the prover can prove membership of elements in the set to verifiers who have the digest.