Jekyll2023-12-06T05:06:37+00:00https://alinush.github.io//feed.xmlAlin TomescuTBD
Alin TomescuScience: Process, not Truth2023-06-02T00:00:00+00:002023-06-02T00:00:00+00:00https://alinush.github.io//2023/06/02/Science-process-not-truth<p class="success"><strong>tl;dr:</strong> <em>“If you thought before that science is certain, well, that’s just an error on your part.”</em> – <a href="http://en.wikipedia.org/wiki/Richard_Feynman">Richard Feynman</a></p>
<!--more-->
<p>I once ran into a <a href="https://www.youtube.com/watch?v=yRxx8pen6JY">video</a> where Neil deGrasse Tyson, in relation to a debate with folks who didn’t “believe”<sup id="fnref:believe" role="doc-noteref"><a href="#fn:believe" class="footnote" rel="footnote">1</a></sup> in global warming nor in evolution, said the following:</p>
<p class="error"><em>“The good thing about science is that it’s true whether or not you believe in it”</em> – <strong>Neil deGrasse Tyson</strong></p>
<p>You can see the short video below:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/yRxx8pen6JY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen=""></iframe>
<h2 id="the-misconception-science-is-not-intrinsically-true">The misconception: Science is not intrinsically “true”</h2>
<p>The way to conceptualize, and popularize<sup id="fnref:neil" role="doc-noteref"><a href="#fn:neil" class="footnote" rel="footnote">2</a></sup>, science is <strong>not</strong> as being “true” or “false.”</p>
<p>Science is a <strong>process</strong> that we engage in to discover truths.
And it often leads us astray<sup id="fnref:galileo" role="doc-noteref"><a href="#fn:galileo" class="footnote" rel="footnote">3</a></sup>$^,$<sup id="fnref:aether" role="doc-noteref"><a href="#fn:aether" class="footnote" rel="footnote">4</a></sup>, which is why the idea of “science being true” is at best misleading and at worst dangerous.</p>
<p>Science is not a <strong>belief</strong> in any sense of the word.
Although many religions might be based on belief, science is an entirely different beast.</p>
<p class="success">Science requires not belief, but <em>experimentation</em>, <em>theory postulation</em> and <em>theory falsification (or refinement)</em>.</p>
<p>Those are fancy words, but here’s an example everyone can understand:</p>
<p><strong>Question:</strong> How might we find out if there is a universal acceleration that falling objects have?
Is it $5\ m/s^2$?
Is it $10\ m/s^2$?</p>
<p><strong>Poor answer:</strong> We could just <strong>blindly</strong> accept whatever answer Neil DeGrasse Tyson gives us. After all <em>“science is true whether you believe it or not,”</em> no?</p>
<p><strong>Better answer:</strong> No.
That is a kind of “scientism” akin to religious belief.
Instead, we could engage in the scientific process:</p>
<ul>
<li><strong>Experimentation</strong>:
<ul>
<li>Drop apples of different sizes several times from the same height</li>
<li>Measure the time they take to fall down</li>
<li>Compute each apple’s acceleration</li>
<li>Observe that all apples, no matter their size, have the same acceleration $g\approx 9.8\ m/s^2$</li>
</ul>
</li>
<li><strong>Theory postulation</strong>:
<ul>
<li>Propose a theory that says, <em>“When dropped on Earth, all objects accelerate with $9.8\ m/s^2$”</em>.</li>
</ul>
</li>
<li><strong>Falsification</strong>:
<ul>
<li>Notice that, for some objects like a light feather, they drop with a smaller acceleration.</li>
<li>Therefore, your proposed theory above is actually false, since it does not hold for feathers.</li>
<li>Refine it as: <em>“When dropped <strong>in a vacuum</strong> on Earth all objects accelerate with $9.8\ m/s^2$”</em><sup id="fnref:feathers" role="doc-noteref"><a href="#fn:feathers" class="footnote" rel="footnote">5</a></sup>.</li>
</ul>
</li>
</ul>
<h3 id="richard-feynman-on-the-uncertainty-of-science">Richard Feynman on the uncertainty of science</h3>
<p>Richard Feynman, a physicist you might know<sup id="fnref:surely" role="doc-noteref"><a href="#fn:surely" class="footnote" rel="footnote">6</a></sup> from his work on <a href="https://en.wikipedia.org/wiki/Quantum_electrodynamics">quantum electrodynamics (QED)</a><sup id="fnref:nobel" role="doc-noteref"><a href="#fn:nobel" class="footnote" rel="footnote">7</a></sup>, explains very beautifully:</p>
<p><strong>What is inherent in science is not “truth” but “uncertainty.”</strong></p>
<p>In one of his famous physics lecture, Feynman beautifully elucidates the <strong>leap of faith</strong> scientists must make when going from a <strong>concrete</strong> scientific experiment to a <strong>general</strong> law that predicts beyond what the experiment tested for.</p>
<p>I include an excerpt below, but I find the <a href="https://www.youtube.com/watch?v=tRxFf3p8Ia8">video</a> to be much more convincing (and entertaining) to watch:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/tRxFf3p8Ia8" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen=""></iframe>
<p>In the video above, Feynman genuinely asks the audience:</p>
<p class="success"><em>Why are we able to extend our laws to regions that we’re not sure?
<br />
How is it possible?
<br />
Why are we so confident??</em></p>
<p>He then explains that extending such general laws is <strong>the thing we must do</strong> if we are to learn anything new beyond the outcome of the concrete experiment.
The <strong>price we pay</strong>, of course, is we (scientists) stand to be proven wrong in the future.
In other words, the nature of scientific laws or theories is they are <strong>uncertain</strong>, subject to be fully-falsified or partially-refined.</p>
<p class="success"><em>It’s not a <strong>mistake</strong> to say that [the law] is true in a region where you haven’t looked yet.
<br />
If you will not say that it’s true in a region that you haven’t looked yet, [then] you don’t know anything!
<br />
[In other words,] If the only laws that you find are those which you just finished observing, then… you can’t make any predictions!
[But] the only utility of the science is to go on and to try and make guesses.</em>
<br />
[…]
<br />
<em>So what we do is <strong>always</strong> to stick our neck out!
<br />
And that of course means that the science is uncertain!</em>
<br />
[…]
<br />
<em>We <strong>always</strong> must make statements about the regions that we haven’t seen, or [else] there’s no use in the whole business.</em></p>
<p>Feynman reiterates on this point to make it stick:</p>
<p class="success"><em>We do not know all the conditions that we need for an experiment!</em>
<br />
[…]
<br />
<em>So in order to have any utility at all to the science…
In order not simply to describe an experiment that’s just been done, we have to propose laws beyond their range.
And there’s nothing wrong with that. That’s the success; that’s the point!
<br />
And, uhh, that makes the science uncertain.
If you thought before that science is certain, well, that’s just an error on your part.</em></p>
<h2 id="the-solution-understanding-and-engaging-with-science">The solution: Understanding and engaging with science</h2>
<p>Please don’t be fooled into thinking “science is true” (whatever that’s supposed to mean).
Such claims are nonsensical.</p>
<p>First, they are nonsensical, because science is a process.
Processes cannot be “true” or “false”; they are just a way of doing things.
Second, because scientific theories are (mostly) <strong>falsifiable</strong>, which is a fancy way of saying there is room to prove them wrong.
(i.e., they might actually be false!)</p>
<p>If one cannot rely on science “to be true”, what should one do instead?</p>
<p>The <strong>first option</strong> is to take a scientific theory and try to falsify it.
For example, Einstein picked Newton’s laws of motion, showed they cannot properly describe motion for objects moving close to the speed of light and generalized them.
And that’s how we got the general theory of relativity and, apparently, GPS on our phones.</p>
<p class="info"><strong>Note:</strong> I look at this as falsifying Netwon’s theories, which were simply not going to work accurately in Einstein’s extreme conditions, while others might look at it as refining them (since Newton’s laws of motion still approximate things very well).</p>
<p>The <strong>second option</strong> is to devise an experiment, make some observations, generalize them into a theory, and see if your theory holds water by checking if it can predict anything useful.
Then, you can go back to the first option and try to falsify your theory.</p>
<p class="success">In other words, engage in science as opposed to <em>“believing in science.”</em></p>
<p>If you must <em>“believe in <strike>science</strike> [the scientific process]”</em>, realize you are simply <strong>deferring to the authority</strong> of other scientists and the soundness of the scientific peer review process.
Yet other scientists are people like you and me.
And we make mistakes, are inherently ignorant or have perverse incentives<sup id="fnref:books" role="doc-noteref"><a href="#fn:books" class="footnote" rel="footnote">8</a></sup>.</p>
<p class="info">I recognize that engaging in the scientific process is a very high bar to meet. I also recognize it’s not clear how to reach consensus faster on scientific theories of high importance such as anthropogenic climate change. But I do know that preaching <em>“science is true; believe science”</em> is a steadfast way of moving from scientific (falsifiable) territory into religious (unfalsifiable) territory. This would defeat the original goals of science: <strong>to ensure we find out when we are wrong and, as a result, get a bit closer to the truth.</strong></p>
<h3 id="digression-on-non-falsifiable-and-thus-non-scientific-theories">Digression on non-falsifiable (and thus non-scientific) theories</h3>
<p>The most widely-accepted philosophy of science comes from Karl Popper, who argued that a theory or statement is scientific only if it can, in principle, be empirically <strong>falsified</strong> (a.k.a., disproven).
In other words, if a theory does not admit any tests that could potentially show it to be false, then the theory is not scientific.</p>
<p>But not all philosophers of science agree with Popper.
Some argue that scientific theories must not only be <em>falsifiable</em> but also <strong>empirically-confirmed</strong> to some degree.
Others argue that scientific theories can be <strong>unfalsifiable in practice</strong> as long as they are part of broader theoretical frameworks that are testable as a whole.
Or, as long as these theories could become falsifiable with future, better technology (e.g., string theory).
In this last case, indirect evidence, logical coherence, and the theory’s ability to explain and predict phenomena is what gets the theory to be viewed as scientific.</p>
<p class="info">In general, celebrating the scientific method does <strong>not</strong> mean we should dispense with all unfalsifiable theories.</p>
<p>A <strong>contrived-but-simple example</strong>: your friend tells you <em>“There exist pink cars with yellow stripes!”</em>
That is an unfalsifiable theory.
Should you ignore everything coming out of their mouth next?</p>
<p>Admittedly, it would be very difficult to convince your friend such cars don’t exist: you’d have to show them that all cars in the universe don’t match this description, which would (likely) take infinite time.
Yet the theory can be shown to be true by a trivial observation of such a car.
And indeed, I know a person with such a car (so I know the theory is true)!<sup id="fnref:idonot" role="doc-noteref"><a href="#fn:idonot" class="footnote" rel="footnote">9</a></sup></p>
<p>The point is, we make unfalsifiable statements (theories) like this all the time and we get along just fine.</p>
<p>Interestingly, <strong>religious theories</strong> (e.g., “there is a God”) fall into the same category of unfalsifiable but potentially-true theories.
In fact, people who claim to have had direct experiences of the divine are firmly-convinced of their veracity.
I no longer believe these theories should be thrown away (i.e., I recognize the limitations of my own form of consciousness).
Instead, I think that the alleged consequences of such theories should be acted upon carefully since, if they are false, there is no way to find out due to their unfalsifiable nature. (In layman terms, don’t burn or kill people because they don’t believe in the same things you do.)</p>
<p>Here’s a more palatable unfalsifiable theory, though harder to formalize.
My wise mother once proposed this theory to me when I was being a smartass about how bad unfalsifiable theories can be.
She said <em>“Okay, how about the theory that <strong>love exists in the world</strong>. Are you gonna throw away that theory too because it’s unfalsifiable?”</em>.
(If you don’t know this theory to be true, then you have other problems and I wish I could give you a warm hug.)</p>
<h2 id="conclusion">Conclusion</h2>
<p>Science is not simply “true”.
It is a process that you could engage in.
It is not an axiom that you take for granted.
It is not an authority that you defer to.</p>
<p>If you don’t have time to engage in science, you can trust-but-verify.
However, there is a risk you’ll be deceived by:</p>
<ul>
<li>“scientific” “experts”</li>
<li>“scientific” evidence</li>
<li>“scientific” truth</li>
<li>claims that “science” shows that $X$ causes $Y$</li>
<li>claims that “studies” show</li>
</ul>
<p>Sentences like these should be critically inspected.</p>
<p class="success"><strong>Thumb rule:</strong> Mentally-remove the word “science” (and its derivatives) from sentences.
See if those sentences still sound convincing.
If they do not, something is being left out and you could investigate.
More and more, the word “science” is used authoritatively without so much as a citation to the relevant scientific work(s).</p>
<p>I do admire deGrasse Tyson’s efforts to bring the scientific process to the masses.
And I’m sure he meant well in (mis)stating that “science is true” (e.g., perhaps he meant to convey, in an entertaining way, his own confidence in the scientific process).</p>
<p>Nonetheless, the conflation of “truth” with “science” and the implication that one should “believe” science worries me.
I, for one, find this borderline-dangerous in our increasingly-polarized society, which is more and more filled with separating beliefs.
Adding science to this list of beliefs would not serve anyone.</p>
<p>I hope this blog post clearly articulated that science is a (fallible) process and must not be blindly trusted.</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:believe" role="doc-endnote">
<p>I write “believe” in quotes because I find the usage of the term “believe” to be over-simplifying when it comes to how one should engage with complex scientific theories like the theory of anthropogenic climate change. In other words, to simply have to pick between <em>“Do you or do you not believe in global warming?”</em> is a disastrous way of getting any clarity on the causes of global warming. A better way might be to ask someone <em>“What evidence is there for anthropogenic climate change and have you taken a close look at it?”</em>. (PS: This blog post is not about the climate change issue.) <a href="#fnref:believe" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:neil" role="doc-endnote">
<p>I do want to recognize Neil deGrasse Tyson’s amazing efforts in popularizing science by speaking about it in a particularly entertaining way. In a way, that’s likely the problem behind his “science is true” claim: he did not carefully balance between entertainment and actual education. <a href="#fnref:neil" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:galileo" role="doc-endnote">
<p>For example, in some less enlightened parts of the word, the “scientific” belief used to be that Earth was in the center of our solar system and that the Sun and stars revolved around it. Galileo Galilei was put into house arrest by the Roman Catholic church for the “heresy” of giving evidence that this theory may be wrong. <a href="#fnref:galileo" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:aether" role="doc-endnote">
<p>Another falsified theory was the <em>“Aether Theory.”</em> Aether was believed to be a medium that filled space and enabled the propagation of light. The famous Michelson-Morley experiment in 1887 failed to detect aether, leading to the eventual acceptance of Einstein’s theory of relativity, which does not require the existence of aether. <a href="#fnref:aether" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:feathers" role="doc-endnote">
<p>This principle of air resistance was famously demonstrated, albeit on the Moon, by astronaut David Scott during the Apollo 15 mission.He dropped a feather and a hammer side by side on the Moon. Due to the absence of air resistance on the Moon, they hit the lunar surface at the same time, providing a visual demonstration of this principle. <a href="#fnref:feathers" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:surely" role="doc-endnote">
<p>You might also know Feynman from his hilarious autobiographical books such as <em>“<a href="https://www.amazon.com/Surely-Feynman-Adventures-Curious-Character/dp/0393316041">Surely you’re joking, Mr. Feynman! (Adventures of a Curious Character)</a>“</em>. <a href="#fnref:surely" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:nobel" role="doc-endnote">
<p>Feynman, Julian Schwinger and Shin’ichirō Tomonaga were jointly awarded the Nobel Prize in 1965 for their contributions to QED. <a href="#fnref:nobel" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:books" role="doc-endnote">
<p>Several books can be (and probably have been) written on the perverse incentives in academia, in scientific peer-review, science funding, etc. <a href="#fnref:books" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:idonot" role="doc-endnote">
<p>I do not. Just trying to make a point. <a href="#fnref:idonot" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Alin Tomescutl;dr: “If you thought before that science is certain, well, that’s just an error on your part.” – Richard FeynmanWhy you should probably never sort your Merkle tree’s leaves2023-02-05T00:00:00+00:002023-02-05T00:00:00+00:00https://alinush.github.io//2023/02/05/Why-you-should-probably-never-sort-your-Merkle-trees-leaves<p class="info"><strong>tl;dr:</strong> …because (1) they are only secure when the tree is correctly-computed (e.g., secure with BFT consensus, but <strong>insecure</strong> in single-server transparency logs), (2) you cannot efficiently insert or delete leaves, and (3) they have worse proof sizes. What does that mean? Never implement one. Stick to <a href="#probably-use-a-merkle-trie">Merkle tries</a> (a.k.a., Merkle prefix trees). Or, if you are a masochist and like to deal with rotations, stick to balanced binary search trees.</p>
<!--more-->
<p hidden="">$$
\def\Adv{\mathcal{A}}
\def\Badv{\mathcal{B}}
\def\vect#1{\mathbf{#1}}
$$</p>
<hr />
<h2 id="problem-statement">Problem statement</h2>
<p>Non-membership proofs in <a href="/2021/02/25/what-is-a-merkle-tree.html">Merkle trees</a> are surprisingly elusive to many people.
The problem statement is very simple:</p>
<p style="margin-left: .3em; border-left: .25em solid black; padding-left: .5em;">
Suppose you have a <strong>server</strong> who wants to authenticate elements of a set $S$ to a <strong>client</strong> without ever sending the whole set to this client.
(For simplicity, let’s assume this is a set of numbers.)
<br /><br />
Specifically, the server first computes a succinct <strong>authentication digest</strong> of the set, denoted by $d$, and sends $d$ to the client.
Then, the server is able to prove either <em>membership</em> or <em>non-membership</em> of an element in the set by sending a succinct <strong>proof</strong> to the <strong>client</strong> which the client can efficiently verify with respect to the digest $d$.
<br /><br />
Design a Merkle tree-based solution for this problem.
</p>
<p>The most popular solution to this problem seems to be to build a Merkle tree whose leaves are sorted.
This, unfortunately, is a rather sub-optimal solution, both from a security and a complexity point of view.</p>
<p>In this blog post, I hope to dispel the myth of the effectiveness of this <em>sorted-leaves</em> Merkle tree scheme.</p>
<h2 id="warm-up-proving-membership-in-merkle-trees">Warm-up: proving membership in Merkle trees</h2>
<p>Recall that it, if we only need to prove membership, it is very easy to solve the problem by building <a href="/2021/02/25/what-is-a-merkle-tree.html">a Merkle tree</a> over all elements in the set and letting the digest be the Merkle root.</p>
<p>For example, here’s how this would look for a particular choice of set $S$. (Original slides <a href="https://docs.google.com/presentation/d/1pbQLuXcI6WjyuNd787ty3U6Qbon41f91e_ixDmgpUgw/edit?usp=sharing">here</a>.)</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-unsorted.png" /></div>
<p>Then, a membership proof would be a <strong>Merkle sibling path</strong> to the proved element’s leaf (i.e., the nodes in <span style="color: #b58900;"><b>yellow</b></span>):</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-memb.png" /></div>
<p>The client can easily verify the proof by computing the hashes along the path from the leaf to the root, and checking that it obtains the same root hash as it has stored locally:</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-memb-verify.png" /></div>
<h2 id="sort-the-leaves-to-add-support-for-non-membership">Sort the leaves to add support for non-membership?</h2>
<p>It seems that many people believe sorting the leaves is the right approach to enable <strong>non</strong>-membership proofs.</p>
<p>This blog post will argue, from three different perspectives, why this is a sub-optimal choice.</p>
<p>Okay! Let say that, instead of the solution <a href="#warm-up-proving-membership-in-merkle-trees">from above</a>, the server does indeed first <strong>sort</strong> the set $S$ as $[2, 3, 6, 7, 9, 11, 13, 14]$ and then computes the Merkle tree:</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-sorted.png" /></div>
<p>Clearly, the server can still prove membership as before: just give a Merkle sibling path to the proved element’s leaf.</p>
<p>But, now, it is also possible to prove non-membership of an element.</p>
<p>For example, we can prove non-membership of $8$ by showing that <em>(a)</em> both $7$ and $9$ are in the tree <strong>and</strong> <em>(b)</em> that they are <strong>adjacent</strong>.
This implies there’s no room where 8 could fit.
Therefore, 8 cannot be in the tree:</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-non-memb.png" /></div>
<p>In other words, the two membership proofs for the adjacent leaves of $7$ and $9$ constitute a <strong>non-membership proof</strong> for 8, which would have to be placed between them (but cannot be since “there’s no room”).</p>
<h2 id="problem-1-security">Problem 1: Security</h2>
<p>Can you spot the security issue? It’s a bit subtle and many people miss it…</p>
<p>Here it is: this scheme is secure <strong>only if</strong> the server correctly computes the Merkle tree over the sorted leaves.</p>
<p>Otherwise, if the server is malicious, it can re-order the leaves and pretend than an element $e$ is <strong>both in</strong> the set <strong>and not in</strong> the set.</p>
<p>For example, the malicious server could compute the tree as follows:</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-incorrectly.png" /></div>
<p>Note that the malicious server left 7 adjacent to 9, so that it can still give <em>what appears to be</em> a valid non-membership proof for 8:</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-attack-non-memb.png" /></div>
<p>At the same time, note that the malicious server inserted a leaf for 8 somewhere else.
As a result, the server can still give <em>what appears to be</em> a valid membership proof for 8:</p>
<div align="center"><img style="width:100%" src="/pictures/sorting-merkle-attack-memb.png" /></div>
<p class="error">This, of course, is very bad: the server was able to prove two <strong>inconsistent</strong> statements about the membership of 8 in the digested set. Put differently, it clearly cannot be that 8 is both in $S$ and not in $S$ at the same time. Therefore, the sorted-leaves Merkle tree is insecure when the server cannot be trusted to produce correct digests (and we’ll define security <a href="#nonmembership-soundness-definitions">below</a>).</p>
<p>In other words, this type of attack completely ruins security: it makes any proof <em>meaningless</em> to the client (e.g., proof that $8\in S$), since it could easily be followed by a contradicting proof (e.g., a contradicting proof that $8\notin S$).</p>
<h3 id="when-would-the-sorted-leaves-merkle-tree-be-secure">When would the sorted-leaves Merkle tree be secure?</h3>
<p>Not all hope is lost. In some settings, it can be reasonable to assume the digest (i.e., Merkle root) was produced correctly.</p>
<p>For example, in distributed consensus settings (a.k.a., in “blockchains”), there is no single server that dictates what the Merkle root of the data is.
Instead, all $n = 3f+1$ servers try to compute the same <em>correct</em> root and vote on it.
Servers who deviate from the correct root are ignored and consensus is reached on the correct one by a subset of $2f+1$ honest servers.</p>
<p>Therefore, in this setting, it is okay to rely on the sorted-leaves Merkle tree construction.
(I’ll still argue <a href="#problem-2-insertions-and-deletions">here</a> and <a href="#problem-3-proof-size-is-sub-optimal">here</a> why you shouldn’t, but from different perspectives.)</p>
<p>Other harmless settings include single-client data outsourcing, where a client sorts & Merkle hashes his own data correctly, and transfers everything but the Merkle root to a malicious server.</p>
<p>Since the client has computed the correct root on his own, the client can rely on the server’s (non)membership proofs.</p>
<p class="error">One thing worth emphasizing is that ad-hoc fixes to the problem of a potentially-incorrect digest are not worth it, especially since one can get a construction that needs no fixing from, e.g., a <a href="#probably-use-a-merkle-trie">Merkle trie</a>. Specifically, it is not worth it to require the server to prove that it correctly sorted the leaves (e.g., via a SNARK). Also, it is not worth it to rely on fraud proofs when one can have provably-correct behavior all the time. Lastly, it is not worth it to probabilistically audit the data structure to see if you can find two incorrectly-sorted leaves. None of these approaches are worth it because there exist more secure Merkle tree constructions like Merkle tries. Plus, these constructions are easier to update and have smaller proof sizes!</p>
<h3 id="nonmembership-soundness-definitions">(Non)membership soundness definitions</h3>
<p>We can formalize the setting in which authenticated set constructions (like the sorted-leaves Merkle tree) are secure.</p>
<p>Specifically, we can define a notion of <em>weak (non)membership soundness</em> that captures the idea that the malicious server must compute the digest correctly:</p>
<p class="info">An authenticated set scheme has <strong>weak (non)membership soundness</strong> if for all (polynomial-time) adversaries $A$, the probability that $A$ outputs a set $S$, an element $e$, and two proofs $\pi$ & $\pi’$ such that, letting $d$ be the (correct) digest of $S$, $\pi$ verifies as a valid membership proof for $e$ (w.r.t. $d$) while $\pi’$ also verifies as a valid <strong>non</strong>-membership proof for $e$ (w.r.t. $d$), is negligible in the security parameter of the scheme.</p>
<p>Notice that the adversary outputs a set of elements from which the correct digest $d$ is computed.</p>
<p>In fact, there is a long line of academic literature on 2-party and 3-party authenticated data structures that rely on this type of weaker soundness definitions (see Papamanthou’s PhD thesis<sup id="fnref:Papa11" role="doc-noteref"><a href="#fn:Papa11" class="footnote" rel="footnote">1</a></sup> for a survey).</p>
<p>Unfortunately, many applications today inherently rely on untrusted publishers who can compute malicious digests of their data.</p>
<p>For example, in key transparency logs such as Certificate Transparency (CT), log servers can present any digest to new clients joining the system. Therefore, in this setting, authenticated data structures (whether sets or not), must satisfy a <strong>stronger</strong> notion of security which allows the adversary to construct the digest maliciously.</p>
<p>In fact, such a <strong>stronger</strong> notion simply requires that the adversary output the digest $d$ directly, which gives the adversary freedom to construct an incorrect one as in <a href="#problem-1-security">our attack above</a>:</p>
<p class="info">An authenticated set scheme has <strong>strong (non)membership soundness</strong> if for all (polynomial-time) adversaries $A$, the probability that $A$ outputs a digest $d$, an element $e$, and two proofs $\pi$ & $\pi’$ such that $\pi$ verifies as a valid membership proof for $e$ (w.r.t. $d$) while $\pi’$ also verifies as a valid <strong>non</strong>-membership proof for $e$ (w.r.t. $d$), is negligible in the security parameter of the scheme.</p>
<p>The moral of the story is to pick a Merkle construction that has this stronger notion of security, unless you are sure that your setting allows for the weaker notion <em>and</em> you stand to benefit from relaxing the security (e.g., perhaps because you get a faster construction).
A good example of this is the <a href="/2020/05/06/kzg-polynomial-commitments.html">KZG</a>-based authenticated dictionary from Ethereum Research<sup id="fnref:Feis20Multi" role="doc-noteref"><a href="#fn:Feis20Multi" class="footnote" rel="footnote">2</a></sup> which has <strong>weak soundness</strong> (as would be defined for dictionaries), but that’s okay since their consensus setting can accommodate it.</p>
<h2 id="problem-2-insertions-and-deletions">Problem 2: Insertions and deletions</h2>
<p>This one is much easier to explain.</p>
<p>Imagine you want to add a new element in your sorted-leaves Merkle tree of size 8.</p>
<p>What if it is smaller than everything else and has to be inserted as the first leaf of the tree?</p>
<p>Then, you would have to completely rehash the entire tree to incorporate this new leaf! This would take $O(n)$ work in a tree of $n$ leaves.</p>
<p>The same problem arises if you’d like to remove the first leaf.</p>
<p class="info">To deal with the slowness of <strong>insertions</strong>, one can take an amortized approach and maintain a <strong>forest</strong> of sorted-leaves Merkle trees, where (1) new leaves are appended to the right of the forest as their own size-1 trees and (2) trees of the same size $2^i$ for any $i \ge 0$ get “merged” together by merge-sorting their leaves and rehashing. One can show this approach has $O(\log{n})$ <strong>amortized</strong> insertion cost. However, such amortized approaches still suffer from $O(n)$ worst-case times and must be de-amortized to bring the worst-case cost down to the amortized cost<sup id="fnref:Eric15" role="doc-noteref"><a href="#fn:Eric15" class="footnote" rel="footnote">3</a></sup>.</p>
<p class="info">On the other hand, dealing with <strong>deletions</strong> can be easier. Specifically, if you do not care about wasted space, then deletions can be done faster by simply marking the leaf as “removed” and trying to garbage-collect as many empty subtrees as you can. Nonetheless, in the worst case, the storage complexity of an $n$-leaf Merkle tree after $O(n)$ deletes remains $O(n)$ (e.g., imagine deleting every even-numbered leaf).</p>
<h2 id="problem-3-proof-size-is-sub-optimal">Problem 3: Proof size is sub-optimal</h2>
<p>The other problem with the sorted leaves construction is that <strong>two</strong> Merkle paths must be given as a non-membership proof.</p>
<p>In the best-case, this can be exactly $\log{n}-1$ hashes, but in the worst case this can be as much as $2\log{n}-2$ hashes (e.g., when one proof is in the left subtree and the other proof is in the right subtree).</p>
<p>This is not so great if proof size is a concern.
It is also not so great when the Merkle tree is stored on disk since it can double the proof reading I/O cost.</p>
<p>Furthermore, actually achieving the best-case proof size complexity in an implementation can be tricky: the developer must efficiently batch the fetching of the two Merkle proofs from disk or memory, taking care never to fetch the same sibling hash twice (or waste I/O).</p>
<h2 id="if-you-really-must-sort-your-leaves">If you really <em>MUST</em> sort your leaves…</h2>
<p>…and you want to maintain strong (non)membership soundness, then there is a simple way to fix your construction.</p>
<p>All you have to do is store, inside each internal node of your tree, the minimum and maximum element in that node’s subtree.</p>
<p>Now, a Merkle proof, whether for membership or not, has to additionally reveal the minimum and maximum’s along the proven path.
Importantly, when hashing up to verify the Merkle proof, the verifier must ensure the revealed leaf and all min/max pairs revealed are consistent and hashed correctly as part of the verification.</p>
<p>This will of course further increase the proof size of your construction.
It will also increase the complexity of implementing the verification procedure, since the min/max ranges have to be incorporated into the hashing and one must check that, for all revealed ranges in the proof, a parent’s range encompasses their child’s range.</p>
<p>Feel free to consider this approach.
You could try reproducing <a href="#problem-1-security">the attack from above</a>.
You’ll see that while you can present one proof, you’ll have difficulty presenting the other because you will not be able to forge the authenticated min/max ranges.
Thus, this construction has <strong>strong (non)membership soundness</strong>.</p>
<h2 id="probably-use-a-merkle-trie">Probably use a Merkle trie</h2>
<p>This deserves its own post, but here are the key reasons <strong>you should probably use a Merkle trie</strong>:</p>
<ul>
<li>Tries over $n$ elements have height $O(\log{n})$, assuming you compress tree paths, which you can! For example, see the CONIKS line of work<sup id="fnref:MBBplus15" role="doc-noteref"><a href="#fn:MBBplus15" class="footnote" rel="footnote">4</a></sup>.</li>
<li>Tries are an intuitive data structure</li>
<li>Tries do not require rotations to keep the tree well-balanced</li>
<li>Merkle tries offer <strong>strong (non)membership soundness</strong></li>
<li>Merkle tries are relatively-easy to implement</li>
</ul>
<p>There are of course some disadvantages too, but I find them negligible:</p>
<ul>
<li>Tries require a bit more hash computations to determine the path of an element in the tree (during insertion, updates, proof verification, etc.)</li>
<li>Tries have some tricky edge-cases when implementing (e.g., inserting two elements whose first $k$ bits collide in an empty trie)</li>
<li>Tries have some tricky edge-cases for proving non-membership
<ul>
<li>e.g., when proving non-membership of $e$, either a leaf exists along $e$’s path but it’s for the wrong element, or no leaf exists at all</li>
<li>This edge case arises in the simplest implementation of tries, which do not ensure the tree is <strong>full</strong> (i.e., <em>fullness</em> means every node other than the leaves has two children)</li>
</ul>
</li>
<li>Tries are vulnerable to adversarial insertions: an adversary can search for a key whose insertion depth will be very large
<ul>
<li>However, to achieve depth $k$, the adversary will have to compute $2^k$ hashes, which gets expensive quickly.</li>
</ul>
</li>
</ul>
<p class="info">In fact, some folks argue that the best trie implementation is via <a href="https://cr.yp.to/critbit.html">critbit trees</a><sup id="fnref:alnoki" role="doc-noteref"><a href="#fn:alnoki" class="footnote" rel="footnote">5</a></sup>.
Unfortunately, I do not know enough about their benefits, especially when Merkleized, but this is probably very much worth exploring.</p>
<h2 id="history-of-non-membership-proofs">History of non-membership proofs</h2>
<p>Kocher<sup id="fnref:Koch98" role="doc-noteref"><a href="#fn:Koch98" class="footnote" rel="footnote">6</a></sup> proves non-revocation of certificates via a sorted-leaves-like approach.
His approach Merkelizes a list of sorted, non-revoked certificate ID ranges.
Specifically, each leaf is a pair $(a, c)$ that says $a$ has been revoked but all certificates $b$ such that $b > a$ and $b < c$ have <strong>not</strong> been revoked.</p>
<p>Thus, one can prove non-revocation of $b$ by revealing the leaf $(a, c)$ that encompasses the non-revoked ID $b \in (a, c)$.
One can also prove revocation of $a$ by revealing the leaf $(a, c)$.</p>
<div align="center"><img style="width:100%" src="/pictures/koch98-crt.png" /></div>
<p class="info">A depiction of the sorted-leaves-like approach from Kocher’s original paper<sup id="fnref:BLL00" role="doc-noteref"><a href="#fn:BLL00" class="footnote" rel="footnote">7</a></sup>.
The set of elements being authenticated here (i.e., revoked certificates) is $S = \{5, 12, 13, 15, 20, 50, 99\}$.</p>
<p>Of course, Kocher’s approach is vulnerable to the same <a href="#problem-1-security">mis-ordering attack we discussed above</a>.
(Furthermore, it also suffers from <a href="#problem-2-insertions-and-deletions">inefficiency of updates</a>.)</p>
<p>Indeed, Buldas et al.<sup id="fnref:BLL00:1" role="doc-noteref"><a href="#fn:BLL00" class="footnote" rel="footnote">7</a></sup> point out the mis-ordering attack and solve the problem by Merkelizing a binary search tree (BST) instead, which they baptize as an <em>authenticated search tree</em>.
However, as far as I could tell, the paper does not describe how to efficiently update such authenticated search trees while keeping them balanced (i.e., solve <a href="#problem-2-insertions-and-deletions">problem 2</a>).</p>
<div align="center"><img style="width:100%" src="/pictures/bll00-ast.png" /></div>
<p class="info">A depiction of the authenticated (binary) search tree approach from Buldas et al.’s original paper<sup id="fnref:BLL00:2" role="doc-noteref"><a href="#fn:BLL00" class="footnote" rel="footnote">7</a></sup>.
The set of elements being authenticated here is $S = \{10, 12, 30, 40, 42, 56, 70, 80\}$.</p>
<p>Fortunately, a few years earlier, Naor and Nissim<sup id="fnref:NN98" role="doc-noteref"><a href="#fn:NN98" class="footnote" rel="footnote">8</a></sup> had proposed an authenticated 2-3 tree construction which did solve the problem of efficient updates, addressing all problems highlighted in this post.
Surprisingly, Naor and Nissim did not point out the mis-ordering attack on Kocher’s work, only the inefficiency of updating it.
Also surprisingly, there are no pictures of trees in their paper <strong>:(</strong></p>
<p>I still find Merkle tries much easier to implement, but I never tried implementing a 2-3 tree.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Hopefully, this post gave you enough context on the problems of this popular sorted-leaves Merkle tree construction.</p>
<p>This leaves me wondering: are there any advantages to sorted-leaves Merkle trees?</p>
<p>The only advantage I see is that MHTs with sorted leaves are easy to describe: just sort the leaves, Merkleize them and prove non-membership of an element $e$ by revealing the two paths to the adjacent leaves that exclude $e$.</p>
<p>However, just because they are easy to describe does <strong>not</strong> mean they are easy to understand.</p>
<p>At least, from the questions and answers I see online, and from conversations with researchers and other engineers, their security caveats are <strong>not</strong> well understood.</p>
<p>First, <a href="https://crypto.stackexchange.com/a/31955/21296">my own answer on StackExchange</a> makes an unfortunate use of the “sorted Merkle tree” terminology to refer to either a binary search tree<sup id="fnref:bst-def" role="doc-noteref"><a href="#fn:bst-def" class="footnote" rel="footnote">9</a></sup>, a trie, or a Sparse Merkle tree (SMT), which actually all have <strong>strong (non)membership soundness</strong>. Even worse, tries and SMTs are not really sorted, since data is typically hashed before being mapped into the trie.</p>
<p>Another <a href="https://crypto.stackexchange.com/questions/83289/proof-of-membership-and-non-membership-in-merkle-tree-hash-tree/83291#83291">StackExchange answer</a> seems to perpetuate the myth that all you need for non-membership security is to sort the leaves, without paying attention to the <strong>weak (non)membership soundness</strong> guarantees of such a construction.</p>
<p>The answer quotes <a href="https://gist.github.com/chris-belcher/eb9abe417d74a7b5f20aabe6bff10de0">this post</a>, where a sorted-leaves Merkle tree solution is described to solve a non-membership problem like <a href="#problem-statement">the one in the intro</a>.
Unfortunately, the answer discards the nuance of the quoted post: there, the original author realized that the leaves could be incorrectly-sorted & resorted to fraud proofs to catch such misbehaviour; i.e., if someone detects a mis-ordered tree, they can easily prove it with two Merkle paths to the out-of-order leaves.</p>
<p>Yet a <strong>much easier</strong> and <strong>cheaper</strong> solution would have been to use an authenticated set with <strong>strong (non)membership soundness</strong> as defined <a href="#nonmembership-soundness-definitions">above</a> (e.g., a <a href="#probably-use-a-merkle-trie">Merkle trie</a>).
This would have simplified the higher-level protocol, since it would have removed the need for fraud proofs, which are clearly less desirable when one can have provably-correct behavior all the time.</p>
<p>Oh well, we live and learn. <strong>Don’t sort your Merkle tree’s leaves</strong>, okay? Use a <a href="#probably-use-a-merkle-trie">Merkle trie</a>.</p>
<p>And, if you somehow find a reason to sort your leaves, please let me know what were the advantages of doing it.
Don’t forget to compare to more secure solutions such as <a href="#probably-use-a-merkle-trie">Merkle tries</a>, which have <strong>strong (non)membership soundness</strong>.</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:Papa11" role="doc-endnote">
<p><strong>Cryptography for Efficiency: New Directions in Authenticated Data Structures</strong>, by Charalampos Papamanthou, 2011, <a href="https://user.eng.umd.edu/~cpap/published/theses/cpap-phd.pdf">[URL]</a> <a href="#fnref:Papa11" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Feis20Multi" role="doc-endnote">
<p><strong>Multi-layer hashmaps for state storage</strong>, by Dankrad Feist, 2020, <a href="https://ethresear.ch/t/multi-layer-hashmaps-for-state-storage/7211/">[URL]</a> <a href="#fnref:Feis20Multi" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Eric15" role="doc-endnote">
<p><strong>Static-to-dynamic tranformations</strong>, by Jeff Erickson, 2015, <a href="http://jeffe.cs.illinois.edu/teaching/datastructures/notes/01-statictodynamic.pdf">[URL]</a> <a href="#fnref:Eric15" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:MBBplus15" role="doc-endnote">
<p><strong>CONIKS: Bringing Key Transparency to End Users</strong>, by Marcela S. Melara and Aaron Blankstein and Joseph Bonneau and Edward W. Felten and Michael J. Freedman, <em>in {24th USENIX Security Symposium (USENIX Security 15)}</em>, 2015, <a href="https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/melara">[URL]</a> <a href="#fnref:MBBplus15" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:alnoki" role="doc-endnote">
<p>Shoutout to <a href="https://github.com/alnoki">Alnoki</a>, the cofounder of <a href="https://www.econialabs.com/">Econia Labs</a> who brought crit-bit trees to my attention. <a href="#fnref:alnoki" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Koch98" role="doc-endnote">
<p><strong>On certificate revocation and validation</strong>, by Kocher, Paul C., <em>in Financial Cryptography</em>, 1998 <a href="#fnref:Koch98" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:BLL00" role="doc-endnote">
<p><strong>Accountable certificate management using undeniable attestations</strong>, by Ahto Buldas and Peeter Laud and Helger Lipmaa, <em>in ACM CCS’00</em>, 2000, <a href="https://doi.org/10.1145%2F352600.352604">[URL]</a> <a href="#fnref:BLL00" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:BLL00:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:BLL00:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:NN98" role="doc-endnote">
<p><strong>Certificate Revocation and Certificate Update</strong>, by Moni Naor and Kobbi Nissim, <em>in 7th USENIX Security Symposium (USENIX Security 98)</em>, 1998, <a href="https://www.usenix.org/conference/7th-usenix-security-symposium/certificate-revocation-and-certificate-update">[URL]</a> <a href="#fnref:NN98" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:bst-def" role="doc-endnote">
<p>Note that a <a href="https://en.wikipedia.org/wiki/Binary_search_tree">binary-search tree (BST)</a> is a tree where all left descendants of a node are smaller than that node & all right descendants of a node are greater than that node. Importantly, trees with sorted leaves are not conceptualized as binary search trees, since their data is stored in the leaves, not in the internal nodes. <a href="#fnref:bst-def" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Alin Tomescutl;dr: …because (1) they are only secure when the tree is correctly-computed (e.g., secure with BFT consensus, but insecure in single-server transparency logs), (2) you cannot efficiently insert or delete leaves, and (3) they have worse proof sizes. What does that mean? Never implement one. Stick to Merkle tries (a.k.a., Merkle prefix trees). Or, if you are a masochist and like to deal with rotations, stick to balanced binary search trees.Pairing-based anonymous credentials and the power of re-randomization2023-01-08T00:00:00+00:002023-01-08T00:00:00+00:00https://alinush.github.io//2023/01/08/Pairing-based-anonymous-credentials-and-the-power-of-re-randomization<p class="info"><strong>tl;dr:</strong> Pointcheval-Sanders (PS) signatures<sup id="fnref:PS16" role="doc-noteref"><a href="#fn:PS16" class="footnote" rel="footnote">1</a></sup> are incredibly powerful: (1) they can sign Pedersen commitments directly and (2) they can be re-randomized together with the signed commitment. This enables very simple schemes for proving yourself anonymously. For example, an authority can give you a PS signature on a commitment of your age and date-of-birth. If you wanna prove to a bar that you are 18 years or older, you can present the bar with a re-randomized signature on the re-randomized commitment together with a zero-knowledge proof that the difference between the current year and your committed birth year is greater than 18.</p>
<p>For more details, see this post on <a href="https://decentralizedthoughts.github.io/2023-01-08-re-rand-cred/">Decentralized Thoughts</a></p>
<!--more-->
<p hidden="">$$
\def\Adv{\mathcal{A}}
\def\Badv{\mathcal{B}}
\def\vect#1{\mathbf{#1}}
$$</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:PS16" role="doc-endnote">
<p><strong>Short Randomizable Signatures</strong>, by Pointcheval, David and Sanders, Olivier, <em>in CT-RSA 2016</em>, 2016 <a href="#fnref:PS16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Alin Tomescutl;dr: Pointcheval-Sanders (PS) signatures[^PS16] are incredibly powerful: (1) they can sign Pedersen commitments directly and (2) they can be re-randomized together with the signed commitment. This enables very simple schemes for proving yourself anonymously. For example, an authority can give you a PS signature on a commitment of your age and date-of-birth. If you wanna prove to a bar that you are 18 years or older, you can present the bar with a re-randomized signature on the re-randomized commitment together with a zero-knowledge proof that the difference between the current year and your committed birth year is greater than 18. For more details, see this post on Decentralized ThoughtsPairings or bilinear maps2022-12-31T20:45:59+00:002022-12-31T20:45:59+00:00https://alinush.github.io//2022/12/31/pairings-or-bilinear-maps<!-- TODO: Add example of pairing (insecure). -->
<p class="info"><strong>tl;dr:</strong> <em>Pairings</em>, or <em>bilinear maps</em>, are a very powerful mathematical tool for cryptography.
Pairings gave us our most succinct zero-knowledge proofs<sup id="fnref:GGPR12e" role="doc-noteref"><a href="#fn:GGPR12e" class="footnote" rel="footnote">1</a></sup>$^,$<sup id="fnref:PGHR13e" role="doc-noteref"><a href="#fn:PGHR13e" class="footnote" rel="footnote">2</a></sup>$^,$<sup id="fnref:Grot16" role="doc-noteref"><a href="#fn:Grot16" class="footnote" rel="footnote">3</a></sup>, our most efficient threshold signatures<sup id="fnref:BLS01" role="doc-noteref"><a href="#fn:BLS01" class="footnote" rel="footnote">4</a></sup>, our first usable identity-based encryption (IBE)<sup id="fnref:BF01" role="doc-noteref"><a href="#fn:BF01" class="footnote" rel="footnote">5</a></sup> scheme, and many other efficient cryptosystems<sup id="fnref:KZG10" role="doc-noteref"><a href="#fn:KZG10" class="footnote" rel="footnote">6</a></sup>.
In this post, I’ll teach you a little about the properties of pairings, their cryptographic applications and their fascinating history.
In fact, by the end of this post, <a href="#history">some of you might want to spend a year or two in jail</a>.</p>
<p class="warning"><strong>Twitter correction:</strong> The <a href="https://twitter.com/alinush407/status/1612518576862408705">original tweet announcing this blog post</a> stated that <em>“<strong>S</strong>NARKs would not be possible without [pairings]”</em>, with the highlighted <strong>S</strong> meant to emphasize the “succinctness” of such SNARKs. However, <a href="#acknowledgements">thanks to several folks on Twitter</a>, I realized this is <strong>not</strong> exactly true and depends on what one means by “succinct.” Specifically, “succinct” SNARKs, in the <em>polylogarithmic proof size</em> sense defined by Gentry and Wichs<sup id="fnref:GW10" role="doc-noteref"><a href="#fn:GW10" class="footnote" rel="footnote">7</a></sup>, exist from a plethora of assumptions, including discrete log<sup id="fnref:BCCplus16" role="doc-noteref"><a href="#fn:BCCplus16" class="footnote" rel="footnote">8</a></sup> or random oracles<sup id="fnref:Mica98" role="doc-noteref"><a href="#fn:Mica98" class="footnote" rel="footnote">9</a></sup>. Furthermore, “succinct” SNARKs, in the sense of $O(1)$ group elements proof size, exist from RSA assumptions too<sup id="fnref:LM18" role="doc-noteref"><a href="#fn:LM18" class="footnote" rel="footnote">10</a></sup>. What pairings do give us, currently, are SNARKs with the smallest, concrete proof sizes (i.e., in # of bytes).</p>
<p hidden="">$$
\def\idt{\mathbb{1}_{\Gr_T}}
\def\msk{\mathsf{msk}}
\def\dsk{\mathsf{dsk}}
\def\mpk{\mathsf{mpk}}
$$</p>
<h2 id="preliminaries">Preliminaries</h2>
<ul>
<li>You are familiar with cyclic groups of prime order (e.g., elliptic curves)</li>
<li>Let \(\idt\) denote the identity element of the group $\Gr_T$</li>
<li>Let $x \randget S$ denote randomly sampling an element $x$ from a set $S$</li>
<li>Recall that $\langle g \rangle = \Gr$ denotes $g$ being a generator of the group $\Gr$</li>
</ul>
<h2 id="definition-of-a-pairing">Definition of a pairing</h2>
<p>A <em>pairing</em>, also known as a <em>bilinear map</em>, is a function $e : \Gr_1 \times \Gr_2 \rightarrow \Gr_T$ between three groups $\Gr_1, \Gr_2$ and $\Gr_T$ of prime order $p$, with generators $g_1 = \langle \Gr_1 \rangle, g_2 = \langle \Gr_2 \rangle$ and $g_T = \langle \Gr_T \rangle$, respectively.</p>
<p>When $\Gr_1 = \Gr_2$, the pairing is called <strong>symmetric</strong>. Otherwise, it is <strong>asymmetric</strong>.</p>
<p>Most importantly, a pairing has two useful properties for cryptography: <em>bilinearity</em> and <em>non-degeneracy</em>.
<!--more--></p>
<h3 id="bilinearity">Bilinearity</h3>
<p><em>Bilinearity</em> requires that, for all $u\in\Gr_1$, $v\in\Gr_2$, and $a,b\in\Zp$:</p>
\[e(u^a, v^b) = e(u, v)^{ab}\]
<p class="warning">For cryptography purposes, this is the <strong>coolest</strong> property. For example, this is what enables useful applications like <a href="#tripartite-diffie-hellman">tripartite Diffie-Hellman</a>.</p>
<h3 id="non-degeneracy">Non-degeneracy</h3>
<p><em>Non-degeneracy</em> requires that:</p>
\[e(g_1, g_2) \ne \idt\]
<p class="info"><strong>Why this property?</strong> We want non-degeneracy because, without it, it is simple (and useless) to define a (degenerate) bilinear map that, for every input, returns $\idt$. Such a map would satisfy bilinearity, but would be completely useless.</p>
<h3 id="efficiency">Efficiency</h3>
<p><em>Efficiency</em> requires that there exists a polynomial-time algorithm in the size of a group element (i.e.; in $\lambda = \log_2{p}$) that can be used to evaluate the pairing $e$ on any inputs.</p>
<details>
<summary><b>Why this requirement?</b> It precludes trivial-but-computationally-intractable pairings. <i>(Click to expand.)</i></summary>
<p style="margin-left: .3em; border-left: .15em solid black; padding-left: .5em;">
For example, let $r$ be a random element in $\Gr_T$.
First, the pairing is defined so that $e(g_1, g_2) = r$.
This way, the pairing satisfies <em>non-degeneracy</em>.
<br /><br />
Second, given $(u,v)\in \Gr_1 \times \Gr_2$, an algorithm could spend exponential time $O(2^\lambda)$ to compute the discrete logs $x = \log_{g_1}{(u)}$ and $y = \log_{g_2}{(v)}$ and return $e(u, v) = e(g_1^x, g_2^y) = r^{xy}$.
This way, the pairing satisfies <em>bilinearity</em> because:
<br /><br />
\begin{align}
e(u^a, v^b)
&= e\left((g_1^x)^a, (g_2^y)^b\right)\\\
&= e\left(g_1^{(ax)}, g_2^{(by)}\right)\\\
&= r^{(ax)\cdot (by)}\\\
&= \left(r^{xy}\right)^{ab}\\\
&= e(u, v)^{ab}
\end{align}
</p>
</details>
<h2 id="history">History</h2>
<p class="warning">This is my limited historical understanding of pairings, mostly informed from <a href="https://www.youtube.com/watch?v=1RwkqZ6JNeo">Dan Boneh’s account in this video</a> and from my own research into the relevant literature. Please email me if you are aware of more history and I can try to incorporate it.</p>
<h3 id="a-mathematician-in-prison">A mathematician in prison</h3>
<p>The history of (cryptographic) pairings begins with a mathematician named <strong>André Weil</strong><sup id="fnref:Wiki22Weil" role="doc-noteref"><a href="#fn:Wiki22Weil" class="footnote" rel="footnote">11</a></sup> who, during World War II, is sent to jail for refusing to serve in the French army.
There, Weil, <em>“managed to convince the liberal-minded prison director to put [him] in an individual cell where [he] was allowed to keep [..] a pen, ink, and paper.”</em></p>
<p>Weil used his newly-acquired tools to define a pairing across two elliptic curve groups.
<strong>However</strong>, what was <strong>very odd</strong> at that time was that Weil put in a lot of effort to make sure his definition of a pairing is <em>computable</em>.
And this extra effort is what made today’s pairing-based cryptography possible<sup id="fnref:danboneh-shimuranote" role="doc-noteref"><a href="#fn:danboneh-shimuranote" class="footnote" rel="footnote">12</a></sup>.</p>
<h3 id="go-to-prison-not-to-university">Go to prison, not to university?</h3>
<p>Funnily, Weil’s time in jail was so productive that he began wondering if he should spend a few months there every year.
Even better, Weil contemplated if he should <strong>recommend to the relevant authorities that every mathematician spend some amount of time in jail.</strong>
Weil writes:</p>
<blockquote>
<p>I’m beginning to think that nothing is more conducive to the abstract sciences than prison.</p>
<p>[…]</p>
<p>My mathematics work is proceeding beyond my wildest hopes, and I am even a bit worried - if it’s only in prison that I work so well, will I have to arrange to spend two or three months locked up every year?</p>
<p>In the meantime, I am contemplating writing a report to the proper authorities, as follows: <em>“To the Director of Scientific Research: Having recently been in a position to discover through personal experience the considerable advantages afforded to pure and disinterested research by a stay in the establishments of the Penitentiary System, I take the liberty of, etc. etc.”</em></p>
</blockquote>
<p>You can read all of this and more in his fascinating autobiography, written from his perspective as a mathematician<sup id="fnref:Weil92" role="doc-noteref"><a href="#fn:Weil92" class="footnote" rel="footnote">13</a></sup>.</p>
<h3 id="from-breaking-cryptography-to-building-cryptography">From breaking cryptography to building cryptography</h3>
<p>Weil’s work was the foundation.
But three more developments were needed for pairing-based cryptography to rise.</p>
<h4 id="first-development-millers-algorithm">First development: Miller’s algorithm</h4>
<p>In 1985, <strong>Victor Miller</strong> writes up a manuscript showing that Weil’s pairing, which actually involves evaluating a polynomial of exponential degree, can in fact be computed efficiently in polynomial time<sup id="fnref:Mill86Short" role="doc-noteref"><a href="#fn:Mill86Short" class="footnote" rel="footnote">14</a></sup>.</p>
<p>In December 1984, Miller gave a talk at IBM about elliptic curve cryptography where he claimed that elliptic curve discrete logs were more difficult to compute than ordinary discrete logs over finite fields<sup id="fnref:miller-talk" role="doc-noteref"><a href="#fn:miller-talk" class="footnote" rel="footnote">15</a></sup>.
Miller was challenged by Manuel Blum, who was in the audience, to back up this claim by giving a <a href="https://en.wikipedia.org/wiki/Reduction_(complexity)">reduction</a>: i.e., showing that an algorithm $B$ for solving discrete log on elliptic curves can be efficiently turned into another algorithm $A$ for solving discrete logs in finite fields.
Such a reduction would imply the problem solved by $B$ (i.e., computing elliptic curve discrete logs) is at least as hard, if not harder, than $A$’s problem (i.e., computing finite field discrete logs).</p>
<p>Miller set out to find a reduction by thinking about the only thing that related an elliptic curve group and a finite field: the Weil pairing.
Funnily, what he quickly realized was that, although the Weil pairing gives a reduction, it’s in the opposite direction: i.e., it turned out an algorithm $A$ for discrete log in finite fields could be efficiently turned into an algorithm $B$ for discrete logs in elliptic curves with the help of the Weil pairing.
This “unwanted” reduction is easy to see.
Since $e(g^a, g) = e(g,g)^a$, solving discrete log on the elliptic curve element $g_a\in \Gr$ is just a matter of solving it on $e(g,g)^a\in \Gr_T$, which is actually a multiplicative subgroup of a finite field (see <a href="#how-do-pairings-actually-work">the inner details of pairings</a>).</p>
<p>This almost showed the opposite of what Miller sought to prove, potentially destroying elliptic curve cryptography, but fortunately the degree of the extension field that the Weil pairing mapped into was too large, making this “unwanted” reduction inefficient and thus not a reduction after all.</p>
<p>This whole affair got Miller interested in seeing if the Weil pairing could be computed efficiently, which led to the discovery of his algorithm.
Interestingly, he submitted this manuscript to FOCS, a top theoretical computer science conference, but the paper got rejected and would not be published until much later in the Journal of Cryptology (according to Miller)<sup id="fnref:alin-where" role="doc-noteref"><a href="#fn:alin-where" class="footnote" rel="footnote">16</a></sup>.</p>
<h4 id="second-development-mov-attack">Second development: MOV attack</h4>
<p>In 1991, <strong>Menezes, Vanstone and Okamoto</strong><sup id="fnref:MVO91" role="doc-noteref"><a href="#fn:MVO91" class="footnote" rel="footnote">17</a></sup> leverage Miller’s efficient algorithm for evaluating the Weil pairing to break the discrete logarithm assumption on certain elliptic curves <strong>in sub-exponential time</strong>.
This was quite amazing since, at that time, no sub-exponential time algorithms were known for elliptic curves.</p>
<p class="info">Their attack, called the <em>MOV attack</em>, mapped an elliptic curve discrete logarithm challenge $g^a\in \Gr$ into a <strong>target group</strong> as $e(g^a, g)=e(g,g)^a \in \Gr_T$ using the pairing.
Since the target group was a subgroup of a finite field $\mathbb{F}_q^{k}$, this allowed the use of faster, sub-exponential time algorithms for computing the discrete log on $e(g,g)^a$.</p>
<h4 id="third-development-jouxs-tripartite-diffie-hellman">Third development: Joux’s tripartite Diffie-Hellman</h4>
<p>So far, pairings only seemed useful for <strong>cryptanalysis.</strong>
No one knew how to use them for building (instead of breaking) cryptography.</p>
<p>This changed in 2000, when <strong>Joux</strong><sup id="fnref:Joux00" role="doc-noteref"><a href="#fn:Joux00" class="footnote" rel="footnote">18</a></sup> used pairings to implement a 1-round key-exchange protocol between three parties, or <a href="#tripartite-diffie-hellman">tripartite Diffie-Hellman</a>.
Previously, such 1-round protocols were only known between two parties while three parties required 2 rounds.</p>
<p>From there, an abundance of new, efficient cryptography started pouring over:</p>
<ul>
<li>BLS (short) signatures<sup id="fnref:BLS01:1" role="doc-noteref"><a href="#fn:BLS01" class="footnote" rel="footnote">4</a></sup></li>
<li>identity-based encryption<sup id="fnref:BF01:1" role="doc-noteref"><a href="#fn:BF01" class="footnote" rel="footnote">5</a></sup></li>
<li>additively-homomorphic encryption with support for one multiplication<sup id="fnref:BGN05" role="doc-noteref"><a href="#fn:BGN05" class="footnote" rel="footnote">19</a></sup></li>
<li>succinct zero-knowledge proofs<sup id="fnref:GGPR12e:1" role="doc-noteref"><a href="#fn:GGPR12e" class="footnote" rel="footnote">1</a></sup></li>
</ul>
<p class="info">An interesting pattern to notice here is how pairings evolved from a <em>cryptanalytic tool</em> used to break cryptosystems, to a <strong>constructive tool</strong> used to build cryptosystems.
Interestingly, the same pattern also arose in the development of lattice-based cryptography.</p>
<h2 id="arithmetic-tricks-with-pairings">Arithmetic tricks with pairings</h2>
<p>There are a few tricks cryptographers often use when dealing with pairings in their proofs of correctness or security of a cryptosystem.</p>
<p>The most obvious trick, <strong>“multiplying in the exponent”</strong>, comes from the bilinearity property.</p>
<p>\begin{align}
e(u^a, v^b) = e(u, v)^{ab}
\end{align}</p>
<p>Bilinearity also implies the following trick:
\begin{align}
e(u, v^b) = e(u, v)^b
\end{align}
Or, alternatively:
\begin{align}
e(u^a, v) = e(u, v)^a
\end{align}</p>
<p>Another trick, which is just an analogous way of defining bilinearity, is:
\begin{align}
e(u, v\cdot w) = e(u, v)\cdot e(u, w)
\end{align}</p>
<p class="info"><strong>Why does this work?</strong> Let $y,z$ denote the discrete logs (w.r.t. $g_2$) of $v$ and $w$, respectively.
Then, we have:
\begin{align}
e(u, v\cdot w)
&= e(u, g_2^y \cdot g_2^z)\\<br />
&= e(u, g_2^{y + z})\\<br />
&= e(u, g_2)^{y + z}\\<br />
&= e(u, g_2)^y \cdot e(u, g_2)^z\\<br />
&= e(u, g_2^y) \cdot e(u, g_2^z)\\<br />
&= e(u, v)\cdot e(u, w)
\end{align}</p>
<p>Or, alternatively:
\begin{align}
e(u, v / w) = \frac{e(u, v)}{e(u, w)}
\end{align}</p>
<h2 id="applications-of-pairings">Applications of pairings</h2>
<h3 id="tripartite-diffie-hellman">Tripartite Diffie-Hellman</h3>
<p>This protocol was introduced by Joux in 2000<sup id="fnref:Joux00:1" role="doc-noteref"><a href="#fn:Joux00" class="footnote" rel="footnote">18</a></sup> and assumes a <strong>symmetric pairing</strong>: i.e., where \(\Gr_1 = \Gr_2 = \langle g\rangle \stackrel{\mathsf{def}}{=} \Gr\).</p>
<p>We have three parties Alice, Bob and Charles with secret keys $a, b$, and $c$ (respectively).
They send each other their public keys $g^a, g^b, g^c$ and agree on a shared secret key $k = e(g, g)^{abc}$.<sup id="fnref:dhe" role="doc-noteref"><a href="#fn:dhe" class="footnote" rel="footnote">20</a></sup></p>
<p>How?</p>
<p>Consider Alice’s point of view.
She gets $g^b$ and $g^c$ from Bob and Charles.
First, she can use her secret $a$ to compute $g^{ab}$.
Second, she can use the pairing to compute $e(g^{ab}, g^c) = e(g, g)^{abc} = k$.</p>
<p>By symmetry, all other players can do the same and agree on the same $k$.</p>
<p class="info">The protocol can be generalized to <a href="#asymmetric-pairings"><strong>a</strong>symmetric pairings</a> too, where $\Gr_1 \neq \Gr_2$.</p>
<h3 id="bls-signatures">BLS signatures</h3>
<p>Boneh, Lynn and Shacham give a very short signature scheme from pairings<sup id="fnref:BLS01:2" role="doc-noteref"><a href="#fn:BLS01" class="footnote" rel="footnote">4</a></sup>, which works as follows:</p>
<ul>
<li>Assume $\Gr_2 = \langle g_2 \rangle$ and that there exists a hash function $H : \{0,1\}^* \rightarrow \Gr_1$ modeled as a random oracle.</li>
<li>The secret key is $s \in \Zp$ while the public key is $\pk = g_2^s \in \Gr_2$.</li>
<li>To sign a message $m$, the signer computes $\sigma = H(m)^s\in \Gr_1$.</li>
<li>To verify a signature $\sigma$ on $m$ under public key $\pk$, one checks if $e(\sigma, g_2) \stackrel{?}{=} e(H(m), \pk)$</li>
</ul>
<p>Notice that correctly-computed signatures will always verify since:
\begin{align}
e(\sigma, g_2) \stackrel{?}{=} e(H(m), \pk) \Leftrightarrow\\<br />
e(H(m)^s, g_2) \stackrel{?}{=} e(H(m), g_2^s) \Leftrightarrow\\<br />
e(H(m), g_2)^s \stackrel{?}{=} e(H(m), g_2)^s \Leftrightarrow\\<br />
e(H(m), g_2) = e(H(m), g_2)
\end{align}</p>
<p>See the BLS paper<sup id="fnref:BLS01:3" role="doc-noteref"><a href="#fn:BLS01" class="footnote" rel="footnote">4</a></sup> for how to prove that no attacker can forge BLS signatures given access to $\pk$ and a signing oracle.</p>
<h4 id="cool-properties-of-bls-signatures">Cool properties of BLS signatures</h4>
<p>BLS signatures are quite amazing:</p>
<ol>
<li>They are one of the <strong>simplest</strong> schemes to implement, given access to an elliptic-curve library.</li>
<li>One can <strong>aggregate</strong> many signatures from different public keys on the same message $m$ into a single <em>multi-signature</em> that continues to verify using just 2 pairings.</li>
<li>One can even <strong>aggregate</strong> such signatures across different messages into a single <em>aggregate signature</em>.
<ul>
<li>However, such aggregate signatures take $n+1$ pairings to verify.</li>
</ul>
</li>
<li>One can easily and efficiently<sup id="fnref:TCZplus20" role="doc-noteref"><a href="#fn:TCZplus20" class="footnote" rel="footnote">21</a></sup> build a <strong>threshold</strong> BLS signature scheme, where any subset of $\ge t$ out of $n$ signers can collaborate to sign a message $m$ but no less than $t$ can ever produce a valid signature.
<ul>
<li>Even better, BLS threshold signatures are <strong>deterministic</strong> and give rise to <em>threshold verifiable random functions (VRFs)</em> which are useful for generating randomness on chain.</li>
</ul>
</li>
<li>One can define very-efficient <strong>blind</strong>-variants of BLS signatures, where the signer can sign a message $m$ without learning the message $m$.</li>
<li>BLS signatures are very <strong>efficient</strong> in practice.
<ul>
<li>As far as I remember, they are the most efficient scheme for (1) multi-signatures, (2) aggregate signatures and (3) threshold signatures</li>
<li>For single-signer BLS, they are slower than Schnorr signatures over non-pairing-friendly curves</li>
</ul>
</li>
</ol>
<p class="warning">If you find yourself confused between the various notions of multi-signatures, aggregate signatures and threshold signatures, see <a href="https://docs.google.com/presentation/d/1G4XGqrBLwqMyDQce_xpPQUEMOK4lFrneuvGYU3MVDsI/edit?usp=sharing">my slides</a>.</p>
<h3 id="identity-based-encryption-ibe">Identity-based encryption (IBE)</h3>
<p>In an IBE scheme, one can encrypt directly to a user-friendly email address (or a phone number), instead of a cumbersome public key which is difficult to remember or type-in correctly.</p>
<p>Boneh and Franklin give a very efficient IBE scheme from pairings<sup id="fnref:BF01:2" role="doc-noteref"><a href="#fn:BF01" class="footnote" rel="footnote">5</a></sup>.</p>
<p>For IBE to work, a trusted third-party (TTP) called a <strong>private key generate (PKG)</strong> must be introduced, who will issue secret keys to users based on their email addresses.
This PKG has a <strong>master secret key (MSK)</strong> $\msk \in \Zp$ with an associated <strong>master public key (MPK)</strong> $\mpk = g_2^s$, where $\langle g_2 \rangle = \Gr_2$.</p>
<p>The $\mpk$ is made public and can be used to encrypt a message to any user given their email address.
Crucially, the PKG must keep the $\msk$ secret.
Otherwise, an attacker who steals it can derive any user’s secret key and decrypt everyone’s messages.</p>
<p class="warning">As you can tell the PKG is a central point of failure: theft of the $\msk$ compromises everyone’s secrecy.
To mitigate against this, the PKG can be decentralized into multiple authorities such that a threshold number of authorities must be compromised in order to steal the $\msk$.</p>
<p>Let $H_1 : \{0,1\}^* \rightarrow \Gr_1^*$ and $H_T : \Gr_T \rightarrow \{0,1\}^n$ be two hash functions modeled as random oracles.
To encrypt an $n$-bit message $m$ to a user with email address $id$, one computes:
\begin{align}
g_{id} &= e(H_1(id), \mpk) \in \Gr_T\\<br />
r &\randget \Zp\\<br />
\label{eq:ibe-ctxt}
c &= \left(g_2^r, m \xor H_T\left(\left(g_{id}\right)^r\right)\right) \in \Gr_2\times \{0,1\}^n
\end{align}</p>
<p>To decrypt, the user with email address $id$ must first obtain their <strong>decryption secret key</strong> $\dsk_{id}$ from the PKG.
For this, we assume the PKG has a way of authenticating the user, before handing them their secret key.
For example this could be done via email.</p>
<p>The PKG computes the user’s decryption secret key as:
\begin{align}
\dsk_{id} = H_1(id)^s \in \Gr_1
\end{align}</p>
<p>Now that the user has their decryption secret key, they can decrypt the ciphertext $c = (u, v)$ from Equation $\ref{eq:ibe-ctxt}$ as:
\begin{align}
m &= v \xor H_T(e(\dsk_{id}, u))
\end{align}</p>
<p>You can see why correctly-encrypted ciphertexts will decrypt successfully, since:
\begin{align}
v \xor H_T(e(\dsk_{id}, u))
&= \left(m \xor H_T\left((g_{id})^r \right)\right) \xor H_T\left(e(H_1(id)^s, g_2^r) \right)\\<br />
&= \left(m \xor H_T\left(e(H_1(id), \mpk )^r \right)\right) \xor H_T\left(e(H_1(id), g_2 )^{rs}\right)\\<br />
&= m \xor \left(H_T\left(e(H_1(id), g_2^s)^r \right) \xor H_T\left(e(H_1(id), g_2 )^{rs}\right)\right)\\<br />
&= m \xor \left(H_T\left(e(H_1(id), g_2 )^{rs}\right) \xor H_T\left(e(H_1(id), g_2 )^{rs}\right)\right)\\<br />
&= m
\end{align}</p>
<p>To see why this scheme is secure under chosen-plaintext attacks, refer to the original paper<sup id="fnref:BF01:3" role="doc-noteref"><a href="#fn:BF01" class="footnote" rel="footnote">5</a></sup>.</p>
<h2 id="how-do-pairings-actually-work">How do pairings actually work?</h2>
<p>Mostly, I have no idea.
How come?
Well, I never really needed to know.
And that’s the beauty of pairings: one can use them in a black-box fashion, with zero awareness of their internals.</p>
<p>Still, let’s take a small peek inside this black box.
Let us consider the popular pairing-friendly <em>BLS12-381</em> curve<sup id="fnref:Edgi22" role="doc-noteref"><a href="#fn:Edgi22" class="footnote" rel="footnote">22</a></sup>, from the family of BLS curves characterized by Barreto, Lynn and Scott<sup id="fnref:BLS02e" role="doc-noteref"><a href="#fn:BLS02e" class="footnote" rel="footnote">23</a></sup>.</p>
<p class="warning"><strong>Public service announcement:</strong>
Some of you might’ve heard about <em>Boneh-Lynn-Shacham (BLS)</em> signatures. Please know that this is a different BLS than the BLS in <em>Barretto-Lynn-Scott</em> curves. Confusingly, both acronyms do share one author, Ben Lynn. (In case this was not confusing enough, wait until you have to work with BLS signatures over BLS12-381 curves.)</p>
<p>For BLS12-381, the three groups $\Gr_1, \Gr_2, \Gr_T$ involved are:</p>
<ul>
<li>The group $\Gr_1$ is a subgroup of an elliptic curve $E(\F_q) = \left\{(x, y) \in (\F_q)^2\ \vert\ y^2 = x^3 + 4 \right\}$ where $\vert\Gr_1\vert = p$</li>
<li>The group $\Gr_2$ is a subgroup of a different elliptic curve $E’(\F_{q^2}) = \left\{(x, y) \in (\F_{q^2})^2\ \vert\ y^2 = x^3 + 4(1+i) \right\}$ where $i$ is the square root of $-1$ and $\vert\Gr_2\vert = p$.</li>
<li>The group $\Gr_T$ are all the $p$th roots of unity in $\F_{q^{k}}$, where $k=12$ is called the <em>embedding degree</em></li>
</ul>
<p>How does the pairing map across these three groups work? Well, the pairing $e(\cdot,\cdot)$ expands to something like:
\begin{align}
\label{eq:pairing-def}
e(u, v) = f_{p, u}(v)^{(q^k - 1)/p}
\end{align}
It’s useful to know that computing a pairing consists of two steps:</p>
<ol>
<li>Evaluating the base $f_{p, u}(v)$, also known as a <strong>Miller loop</strong>, in honor of <a href="#history">Victor Miller’s work</a></li>
<li>Raising this base to the constant exponent $(q^k - 1)/p$, also known as a <strong>final exponentiation</strong>.
<ul>
<li>This step is a few times more expensive than the first</li>
</ul>
</li>
</ol>
<p>For more on the internals, see other resources<sup id="fnref:Cost12" role="doc-noteref"><a href="#fn:Cost12" class="footnote" rel="footnote">24</a></sup>$^,$<sup id="fnref:GPS08" role="doc-noteref"><a href="#fn:GPS08" class="footnote" rel="footnote">25</a></sup>$^,$<sup id="fnref:Mene05" role="doc-noteref"><a href="#fn:Mene05" class="footnote" rel="footnote">26</a></sup>.</p>
<h2 id="implementing-pairing-based-crypto">Implementing pairing-based crypto</h2>
<p>This section discusses various implementation-level details that practitioners can leverage to speed up their implementations.</p>
<h3 id="use-asymmetric-pairings">Use asymmetric pairings!</h3>
<p>The pairing over BLS12-381 is <em><strong>a</strong>symmetric</em>: i.e., $\Gr_1 \ne \Gr_2$ are two <strong>different</strong> groups (of the same order $p$). However, there also exist <strong>symmetric</strong> pairings where $\Gr_1 = \Gr_2$ are the same group.</p>
<p>Unfortunately, <em>“such symmetric pairings only exist on supersingular curves, which places a heavy restriction on either or both of the underlying efficiency and security of the protocol”</em><sup id="fnref:BCMplus15e" role="doc-noteref"><a href="#fn:BCMplus15e" class="footnote" rel="footnote">27</a></sup>.
In other words, such supersingular curves are not as efficient at the same security level as the curves used in <strong>a</strong>symmetric pairings.</p>
<p>Therefore, practitioners today, as far as I am aware, exclusively rely on <strong>a</strong>symmetric pairings due to their higher efficiency when the security level is kept the same.</p>
<h3 id="bls12-381-performance">BLS12-381 performance</h3>
<p>I will give a few key performance numbers for the BLS12-381 curve implemented in Filecoin’s (<a href="https://github.com/filecoin-project/blstrs">blstrs</a> Rust wrapper around the popular <a href="https://github.com/supranational/blst">blst</a> library.
These microbenchmarks were run on a 10-core 2021 Apple M1 Max using <code class="language-plaintext highlighter-rouge">cargo bench</code>.</p>
<h4 id="pairing-computation-times">Pairing computation times</h4>
<!--
alinush@MacBook [~/repos/blstrs] (master %) $ cargo +nightly bench -- pairing_
running 4 tests
test bls12_381::bench_pairing_final_exponentiation ... bench: 276,809 ns/iter (+/- 1,911)
test bls12_381::bench_pairing_full ... bench: 484,718 ns/iter (+/- 2,510)
test bls12_381::bench_pairing_g2_preparation ... bench: 62,395 ns/iter (+/- 4,161)
test bls12_381::bench_pairing_miller_loop ... bench: 148,534 ns/iter (+/- 1,203)
-->
<p>As explained in Eq. \ref{eq:pairing-def}, a pairing involves two steps:</p>
<ul>
<li>a Miller loop computation
<ul>
<li>210 microseconds</li>
</ul>
</li>
<li>a final exponentiation
<ul>
<li>276 microseconds</li>
</ul>
</li>
</ul>
<p>Therefore, a pairing takes around 486 microseconds (i.e., the sum of the two).</p>
<h4 id="exponentiation-times">Exponentiation times</h4>
<p class="warning">The $\Gr_T$ microbenchmarks were done by slightly-modifying the <code class="language-plaintext highlighter-rouge">blstrs</code> benchmarking code <a href="https://github.com/filecoin-project/blstrs/blob/e70aff6505fb6f87f9a13e409c080995bd0f244e/benches/bls12_381/ec.rs#L10">here</a>.
(See the HTML comments of this page for those modifications.)</p>
<!--
alinush@MacBook [~/repos/blstrs] (master *%) $ git diff
diff --git a/benches/bls12_381/ec.rs b/benches/bls12_381/ec.rs
index 639bcad..8dcec20 100644
--- a/benches/bls12_381/ec.rs
+++ b/benches/bls12_381/ec.rs
@@ -167,3 +167,34 @@ mod g2 {
});
}
}
+
+mod gt {
+ use rand_core::SeedableRng;
+ use rand_xorshift::XorShiftRng;
+
+ use blstrs::*;
+ use ff::Field;
+ use group::Group;
+
+ #[bench]
+ fn bench_gt_mul_assign(b: &mut ::test::Bencher) {
+ const SAMPLES: usize = 1000;
+
+ let mut rng = XorShiftRng::from_seed([
+ 0x59, 0x62, 0xbe, 0x5d, 0x76, 0x3d, 0x31, 0x8d, 0x17, 0xdb, 0x37, 0x32, 0x54, 0x06,
+ 0xbc, 0xe5,
+ ]);
+
+ let v: Vec<(Gt, Scalar)> = (0..SAMPLES)
+ .map(|_| (Gt::random(&mut rng), Scalar::random(&mut rng)))
+ .collect();
+
+ let mut count = 0;
+ b.iter(|| {
+ let mut tmp = v[count].0;
+ tmp *= v[count].1;
+ count = (count + 1) % SAMPLES;
+ tmp
+ });
+ }
+}
alinush@MacBook [~/repos/blstrs] (master *%) $ cargo +nightly bench -- mul_assign
Compiling blstrs v0.6.1 (/Users/alinush/repos/blstrs)
Finished bench [optimized] target(s) in 0.75s
Running unittests src/lib.rs (target/release/deps/blstrs-349120dc60ef3711)
running 2 tests
test fp::tests::test_fp_mul_assign ... ignored
test scalar::tests::test_scalar_mul_assign ... ignored
test result: ok. 0 passed; 0 failed; 2 ignored; 0 measured; 115 filtered out; finished in 0.00s
Running benches/blstrs_benches.rs (target/release/deps/blstrs_benches-a6732e3e4e5c6a4d)
running 4 tests
test bls12_381::ec::g1::bench_g1_mul_assign ... bench: 72,167 ns/iter (+/- 1,682)
test bls12_381::ec::g2::bench_g2_mul_assign ... bench: 136,184 ns/iter (+/- 1,300)
test bls12_381::ec::gt::bench_gt_mul_assign ... bench: 497,330 ns/iter (+/- 7,802)
test bls12_381::scalar::bench_scalar_mul_assign ... bench: 14 ns/iter (+/- 0)
test result: ok. 0 passed; 0 failed; 0 ignored; 4 measured; 21 filtered out; finished in 5.30s
-->
<ul>
<li>$\Gr_1$ exponentiations are the fastest
<ul>
<li>72 microseconds</li>
</ul>
</li>
<li>$\Gr_2$ exponentiations are around 2$\times$ slower
<ul>
<li>136 microseconds</li>
</ul>
</li>
<li>$\Gr_T$ exponentiations are around 3.5$\times$ slower than in $\Gr_2$
<ul>
<li>500 microseconds</li>
</ul>
</li>
</ul>
<p class="info"><strong>Note:</strong> These benchmarks pick the exponentiated base randomly and do <strong>not</strong> perform any precomputation on it, which would speed up these times by $2\times$-$4\times$.</p>
<h4 id="multi-exponentiations">Multi-exponentiations</h4>
<p>This is a well-known optimization that I’m including for completeness.</p>
<p>Specifically, many libraries allow you to compute a product $\prod_{0 < i < k} \left(g_i\right)^{x_i}$ of $k$ exponentiations much faster than individually computing the $k$ exponentiations and aggregating their product.</p>
<p>For example, <a href="https://github.com/filecoin-project/blstrs">blstrs</a> seems to be incredibly fast in this regard:</p>
<!--
running 4 tests
test bls12_381::bench_g1_multi_exp ... bench: 760,554 ns/iter (+/- 47,355)
test bls12_381::bench_g1_multi_exp_naive ... bench: 18,575,716 ns/iter (+/- 42,688)
test bls12_381::bench_g2_multi_exp ... bench: 1,876,416 ns/iter (+/- 58,743)
test bls12_381::bench_g2_multi_exp_naive ... bench: 35,272,720 ns/iter (+/- 266,279)
-->
<ul>
<li>a size-256 multi-exponentiation in $\Gr_1$
<ul>
<li>takes 760 microseconds in total, or 3 microseconds per exponentiation!</li>
<li>done naively, it would take 18.5 milliseconds in total, which is $24\times$ longer</li>
</ul>
</li>
<li>a size-256 multi-exponentiation in $\Gr_2$
<ul>
<li>takes 1.88 milliseconds in total, or 7.33 microseconds per exponentiation!</li>
<li>done naively, it would take 35.3 milliseconds, which is $18.8\times$ longer</li>
</ul>
</li>
</ul>
<h4 id="group-element-sizes">Group element sizes</h4>
<ul>
<li>$\Gr_1$ group elements are the smallest
<ul>
<li>e.g., 48 bytes for BLS12-381 or 32 bytes for BN254 curves<sup id="fnref:BN06Pair" role="doc-noteref"><a href="#fn:BN06Pair" class="footnote" rel="footnote">28</a></sup></li>
</ul>
</li>
<li>$\Gr_2$ group elements are 2$\times$ larger
<ul>
<li>e.g., 96 bytes on BLS12-381</li>
</ul>
</li>
<li>$\Gr_T$ elements are 12$\times$ larger
<ul>
<li>In general, for a pairing-friendly curve with <em>embedding degree</em> $k$, they are $k$ times larger</li>
</ul>
</li>
</ul>
<h4 id="other-operations">Other operations</h4>
<ul>
<li>$\Gr_1$ multiplications (recall we are using multiplicative notation for groups, not additive notation)
<ul>
<li>Normal: 565 nanoseconds (when both points are in projective $(X, Y)$ coordinates)</li>
<li>Mixed: 438 nanoseconds (when first point is in projective coordinates, second is in affine $(X, Y, Z)$ coordinates)
<ul>
<li>Faster, because saves one projective-to-affine conversion</li>
</ul>
</li>
</ul>
</li>
<li>$\Gr_2$ multiplications
<ul>
<li>Normal: 1,484 nanoseconds</li>
<li>Mixed: 1,095 nanoseconds</li>
</ul>
</li>
<li>Hashing to $\Gr_1$ takes around 50 microseconds (not accounting for the extra time required to hash down larger messages using SHA2-256)</li>
</ul>
<h4 id="switching-between-gr_1-and-gr_2">Switching between $\Gr_1$ and $\Gr_2$</h4>
<p>When designing a pairing-based cryptographic protocol, you will want to carefully pick what to use $\Gr_1$ and what to use $\Gr_2$ for.</p>
<p>For example, in BLS signatures, if you want small signatures, then you would compute the signature $\sigma = H(m)^s \in \Gr_1$ and settle for a slightly-larger public key be in $\Gr_2$.
On the other hand, if you wanted to minimize public key size, then you would let it be in $\Gr_1$ while taking slightly longer to compute the signature in $\Gr_2$.</p>
<p class="warning">Other things will also influence how you use $\Gr_1$ and $\Gr_2$, such as the existence of an isomorphism $\phi : \Gr_2 \rightarrow \Gr_1$ or the ability to hash uniformly into these groups.
In fact, the existence of such an isomorphism separates between two types of asymmetric pairings: Type 2 and Type 3 (see <em>Galbraith et al.</em><sup id="fnref:GPS08:1" role="doc-noteref"><a href="#fn:GPS08" class="footnote" rel="footnote">25</a></sup> for more information on the different types of pairings)</p>
<h4 id="comparison-to-non-pairing-friendly-elliptic-curves">Comparison to non-pairing-friendly elliptic curves</h4>
<p>When compared to an elliptic curve that does not admit pairings, pairing-friendly elliptic curves are around two times slower.</p>
<p>For example, the popular prime-order elliptic curve group <a href="https://ristretto.group/">Ristretto255</a> offers:</p>
<!--
ristretto255/basepoint_mul
time: [10.259 µs 10.263 µs 10.267 µs]
ristretto255/point_mul time: [40.163 µs 40.187 µs 40.212 µs]
-->
<ul>
<li>$\approx 2\times$ faster exponentiations of 40 microseconds
<ul>
<li>which can be sped up to 10 microseconds, using precomputation when the exponentiation base is fixed</li>
</ul>
</li>
<li>32 byte group element sizes</li>
</ul>
<h3 id="multi-pairings">Multi-pairings</h3>
<p>If you recall how a pairing actually works (see Eq. $\ref{eq:pairing-def}$), you’ll notice the following optimization:</p>
<p>Whenever, we have to compute the product of $n$ pairings, we can first compute the $n$ Miller loops and do a single final exponentiation instead of $n$.
This drastically reduces the pairing computation time in many applications.
\begin{align}
\prod_i e(u_i, v_i)
&= \prod_i \left(f_{p, u_i}(v_i)^{(q^k - 1)/p}\right)\\<br />
&= \left(\prod_i f_{p, u_i}(v_i)\right)^{(q^k - 1)/p}
\end{align}</p>
<h2 id="conclusion">Conclusion</h2>
<p>This blog post was supposed to be just a short summary of the <a href="#definition-of-a-pairing">three properties of pairings</a>: bilinearity, non-degeneracy and efficiency.</p>
<p>Unfortunately, I felt compelled to discuss their <a href="#history">fascinating history</a>.
And I couldn’t let you walk away without seeing a few powerful <a href="#applications-of-pairings">cryptographic applications of pairings</a>.</p>
<p>After that, I realized practitioners who implement pairing-based cryptosystems might benefit from knowing a little about their <a href="#how-do-pairings-actually-work">internal workings</a>, since some of these details can be leveraged to speed up <a href="#implementation-details">implementations</a>.</p>
<h2 id="acknowledgements">Acknowledgements</h2>
<p>I would like to thank Dan Boneh for helping me clarify and contextualize the history around Weil, as well as for <a href="https://www.youtube.com/watch?v=1RwkqZ6JNeo">his 2015 Simons talk</a>, which inspired me to do a little more research and write this historical account.</p>
<p>Big thanks to:</p>
<ul>
<li><a href="https://twitter.com/cronokirby">Lúcás Meier</a>, <a href="https://twitter.com/zkproofs">Pratyush Mishra</a>, <a href="https://twitter.com/rel_zeta_tech">Ariel Gabizon</a> and <a href="https://twitter.com/dariofiore0">Dario Fiore</a> for their enlightening points on what “succinct” (S) stands for in <strong>S</strong>NARKs<sup id="fnref:GW10:1" role="doc-noteref"><a href="#fn:GW10" class="footnote" rel="footnote">7</a></sup> and for reminding me that SNARKs with $O(1)$ group elements proof size exist from RSA assumptions<sup id="fnref:LM18:1" role="doc-noteref"><a href="#fn:LM18" class="footnote" rel="footnote">10</a></sup>.</li>
<li><a href="https://twitter.com/swasilyev">Sergey Vasilyev</a> for pointing out typos in the BLS12-381 elliptic curve definitions.</li>
<li><a href="https://twitter.com/BlakeMScurr">@BlakeMScurr</a> for pointing out an incorrect reference to Joux’s work<sup id="fnref:Joux00:2" role="doc-noteref"><a href="#fn:Joux00" class="footnote" rel="footnote">18</a></sup>.</li>
<li><a href="https://twitter.com/conradoplg">Conrado Guovea</a> for pointing me to Victor Miller’s account of how he developed his algorithm for evaluating the Weil pairing (discussed <a href="#first-development-millers-algorithm">here</a>).</li>
<li><a href="https://twitter.com/ChrisPeikert">Chris Peikert</a> for pointing out that there are plenty-fast IBE schemes out there that do not rely on pairings<sup id="fnref:DLP14e" role="doc-noteref"><a href="#fn:DLP14e" class="footnote" rel="footnote">29</a></sup>.</li>
</ul>
<p><strong>PS:</strong> Twitter threads are a pain to search through, so if I missed acknowledging your contribution, please kindly let me know.</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:GGPR12e" role="doc-endnote">
<p><strong>Quadratic Span Programs and Succinct NIZKs without PCPs</strong>, by Rosario Gennaro and Craig Gentry and Bryan Parno and Mariana Raykova, <em>in Cryptology ePrint Archive, Paper 2012/215</em>, 2012, <a href="https://eprint.iacr.org/2012/215">[URL]</a> <a href="#fnref:GGPR12e" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:GGPR12e:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:PGHR13e" role="doc-endnote">
<p><strong>Pinocchio: Nearly Practical Verifiable Computation</strong>, by Bryan Parno and Craig Gentry and Jon Howell and Mariana Raykova, <em>in Cryptology ePrint Archive, Paper 2013/279</em>, 2013, <a href="https://eprint.iacr.org/2013/279">[URL]</a> <a href="#fnref:PGHR13e" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Grot16" role="doc-endnote">
<p><strong>On the Size of Pairing-Based Non-interactive Arguments</strong>, by Groth, Jens, <em>in Advances in Cryptology – EUROCRYPT 2016</em>, 2016 <a href="#fnref:Grot16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:BLS01" role="doc-endnote">
<p><strong>Short Signatures from the Weil Pairing</strong>, by Boneh, Dan and Lynn, Ben and Shacham, Hovav, <em>in Advances in Cryptology — ASIACRYPT 2001</em>, 2001 <a href="#fnref:BLS01" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:BLS01:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:BLS01:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:BLS01:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:BF01" role="doc-endnote">
<p><strong>Identity-Based Encryption from the Weil Pairing</strong>, by Boneh, Dan and Franklin, Matt, <em>in Advances in Cryptology — CRYPTO 2001</em>, 2001 <a href="#fnref:BF01" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:BF01:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:BF01:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a> <a href="#fnref:BF01:3" class="reversefootnote" role="doc-backlink">↩<sup>4</sup></a></p>
</li>
<li id="fn:KZG10" role="doc-endnote">
<p><strong>Constant-Size Commitments to Polynomials and Their Applications</strong>, by Kate, Aniket and Zaverucha, Gregory M. and Goldberg, Ian, <em>in ASIACRYPT’10</em>, 2010 <a href="#fnref:KZG10" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:GW10" role="doc-endnote">
<p><strong>Separating Succinct Non-Interactive Arguments From All Falsifiable Assumptions</strong>, by Craig Gentry and Daniel Wichs, <em>in Cryptology ePrint Archive, Report 2010/610</em>, 2010, <a href="https://eprint.iacr.org/2010/610">[URL]</a> <a href="#fnref:GW10" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:GW10:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:BCCplus16" role="doc-endnote">
<p><strong>Efficient Zero-Knowledge Arguments for Arithmetic Circuits in the Discrete Log Setting</strong>, by Jonathan Bootle and Andrea Cerulli and Pyrros Chaidos and Jens Groth and Christophe Petit, <em>in Cryptology ePrint Archive, Report 2016/263</em>, 2016, <a href="https://eprint.iacr.org/2016/263">[URL]</a> <a href="#fnref:BCCplus16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Mica98" role="doc-endnote">
<p><strong>Computationally-Sound Proofs</strong>, by Silvio Micali, <em>in Logic Colloquium ‘95: Proceedings of the Annual European Summer Meeting of the Association of Symbolic Logic</em>, 1998, <a href="https://projecteuclid.org/ebooks/lecture-notes-in-logic/Computationally-Sound-Proofs/chapter/Computationally-Sound-Proofs/lnl/1235415908?tab=ChapterArticleLink">[URL]</a> <a href="#fnref:Mica98" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:LM18" role="doc-endnote">
<p><strong>Subvector Commitments with Application to Succinct Arguments</strong>, by Russell W.F. Lai and Giulio Malavolta, <em>in Cryptology ePrint Archive, Report 2018/705</em>, 2018, <a href="https://eprint.iacr.org/2018/705">[URL]</a> <a href="#fnref:LM18" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:LM18:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:Wiki22Weil" role="doc-endnote">
<p><strong>André Weil — Wikipedia, The Free Encyclopedia</strong>, by Wikipedia contributors, 2022, <a href="https://en.wikipedia.org/w/index.php?title=Andr%C3%A9_Weil&oldid=1124211220">[URL]</a> <a href="#fnref:Wiki22Weil" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:danboneh-shimuranote" role="doc-endnote">
<p>Thanks to Dan Boneh, who contrasted Weil’s definition with a different one by Shimura from his classic book on modular forms. While Shimura’s definition makes it much easier to prove all the properties of the pairing, it defines a pairing of order $n$ as a <strong>sum of $n$ points of order $n^2$</strong>. This makes it hopelessly non-computable. Weil’s definition, on the other hand, involves an evaluation of a very concrete function – there are no exponential-sized sums – but requires much more work to prove all its pairing properties. <a href="#fnref:danboneh-shimuranote" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Weil92" role="doc-endnote">
<p><strong>The Apprenticeship of a Mathematician</strong>, by Weil, Andre, 1992, <a href="https://books.google.ro/books?id=73REHmJ9JNUC">[URL]</a> <a href="#fnref:Weil92" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Mill86Short" role="doc-endnote">
<p><strong>Short Programs for functions on Curves</strong>, by Victor S. Miller, 1986, <a href="https://crypto.stanford.edu/miller">[URL]</a> <a href="#fnref:Mill86Short" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:miller-talk" role="doc-endnote">
<p>Miller tells this story himself in <a href="https://www.youtube.com/watch?v=yK5fYfn6HJg&t=2901s">a talk he gave at Microsoft Research</a> on October 10th, 2010. <a href="#fnref:miller-talk" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:alin-where" role="doc-endnote">
<p>I am unable to find any trace of Miller’s published work on this beyond the manuscript Boneh published in<sup id="fnref:Mill86Short:1" role="doc-noteref"><a href="#fn:Mill86Short" class="footnote" rel="footnote">14</a></sup>. Any pointers would be appreciated. <a href="#fnref:alin-where" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:MVO91" role="doc-endnote">
<p><strong>Reducing Elliptic Curve Logarithms to Logarithms in a Finite Field</strong>, by Menezes, Alfred and Vanstone, Scott and Okamoto, Tatsuaki, <em>in ACM STOC</em>, 1991, <a href="http://doi.acm.org/10.1145/103418.103434">[URL]</a> <a href="#fnref:MVO91" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Joux00" role="doc-endnote">
<p><strong>A One Round Protocol for Tripartite Diffie–Hellman</strong>, by Joux, Antoine, <em>in Algorithmic Number Theory</em>, 2000 <a href="#fnref:Joux00" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:Joux00:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:Joux00:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:BGN05" role="doc-endnote">
<p><strong>Evaluating 2-DNF Formulas on Ciphertexts</strong>, by Boneh, Dan and Goh, Eu-Jin and Nissim, Kobbi, <em>in Theory of Cryptography</em>, 2005 <a href="#fnref:BGN05" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:dhe" role="doc-endnote">
<p>Typically, there will be some key-derivation function $\mathsf{KDF}$ used to derive the key as $k = \mathsf{KDF}(e(g,g)^{abc})$. <a href="#fnref:dhe" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:TCZplus20" role="doc-endnote">
<p><strong>Towards Scalable Threshold Cryptosystems</strong>, by Alin Tomescu and Robert Chen and Yiming Zheng and Ittai Abraham and Benny Pinkas and Guy Golan Gueta and Srinivas Devadas, <em>in IEEE S\&P’20</em>, 2020 <a href="#fnref:TCZplus20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Edgi22" role="doc-endnote">
<p><strong>BLS12-381 For The Rest Of Us</strong>, by Ben Edgington, 2022, <a href="https://hackmd.io/@benjaminion/bls12-381">[URL]</a> <a href="#fnref:Edgi22" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:BLS02e" role="doc-endnote">
<p><strong>Constructing Elliptic Curves with Prescribed Embedding Degrees</strong>, by Paulo S. L. M. Barreto and Ben Lynn and Michael Scott, <em>in Cryptology ePrint Archive, Paper 2002/088</em>, 2002, <a href="https://eprint.iacr.org/2002/088">[URL]</a> <a href="#fnref:BLS02e" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Cost12" role="doc-endnote">
<p><strong>Pairings for beginners</strong>, by Craig Costello, 2012, <a href="https://static1.squarespace.com/static/5fdbb09f31d71c1227082339/t/5ff394720493bd28278889c6/1609798774687/PairingsForBeginners.pdf">[URL]</a> <a href="#fnref:Cost12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:GPS08" role="doc-endnote">
<p><strong>Pairings for cryptographers</strong>, by Steven D. Galbraith and Kenneth G. Paterson and Nigel P. Smart, <em>in Discrete Applied Mathematics</em>, 2008, <a href="http://www.sciencedirect.com/science/article/pii/S0166218X08000449">[URL]</a> <a href="#fnref:GPS08" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:GPS08:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:Mene05" role="doc-endnote">
<p><strong>An Introduction to Pairing-Based Cryptography</strong>, by Alfred Menezes, 2005, <a href="https://www.math.uwaterloo.ca/~ajmeneze/publications/pairings.pdf">[URL]</a> <a href="#fnref:Mene05" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:BCMplus15e" role="doc-endnote">
<p><strong>Subgroup security in pairing-based cryptography</strong>, by Paulo S. L. M. Barreto and Craig Costello and Rafael Misoczki and Michael Naehrig and Geovandro C. C. F. Pereira and Gustavo Zanon, <em>in Cryptology ePrint Archive, Paper 2015/247</em>, 2015, <a href="https://eprint.iacr.org/2015/247">[URL]</a> <a href="#fnref:BCMplus15e" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:BN06Pair" role="doc-endnote">
<p><strong>Pairing-Friendly Elliptic Curves of Prime Order</strong>, by Barreto, Paulo S. L. M. and Naehrig, Michael, <em>in Selected Areas in Cryptography</em>, 2006 <a href="#fnref:BN06Pair" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:DLP14e" role="doc-endnote">
<p><strong>Efficient Identity-Based Encryption over NTRU Lattices</strong>, by Léo Ducas and Vadim Lyubashevsky and Thomas Prest, <em>in Cryptology ePrint Archive, Paper 2014/794</em>, 2014, <a href="https://eprint.iacr.org/2014/794">[URL]</a> <a href="#fnref:DLP14e" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Alin Tomescutl;dr: Pairings, or bilinear maps, are a very powerful mathematical tool for cryptography. Pairings gave us our most succinct zero-knowledge proofs[^GGPR12e]$^,$[^PGHR13e]$^,$[^Grot16], our most efficient threshold signatures[^BLS01], our first usable identity-based encryption (IBE)[^BF01] scheme, and many other efficient cryptosystems[^KZG10]. In this post, I’ll teach you a little about the properties of pairings, their cryptographic applications and their fascinating history. In fact, by the end of this post, some of you might want to spend a year or two in jail. Twitter correction: The original tweet announcing this blog post stated that “SNARKs would not be possible without [pairings]”, with the highlighted S meant to emphasize the “succinctness” of such SNARKs. However, thanks to several folks on Twitter, I realized this is not exactly true and depends on what one means by “succinct.” Specifically, “succinct” SNARKs, in the polylogarithmic proof size sense defined by Gentry and Wichs[^GW10], exist from a plethora of assumptions, including discrete log[^BCCplus16] or random oracles[^Mica98]. Furthermore, “succinct” SNARKs, in the sense of $O(1)$ group elements proof size, exist from RSA assumptions too[^LM18]. What pairings do give us, currently, are SNARKs with the smallest, concrete proof sizes (i.e., in # of bytes). $$ \def\idt{\mathbb{1}_{\Gr_T}} \def\msk{\mathsf{msk}} \def\dsk{\mathsf{dsk}} \def\mpk{\mathsf{mpk}} $$ Preliminaries You are familiar with cyclic groups of prime order (e.g., elliptic curves) Let \(\idt\) denote the identity element of the group $\Gr_T$ Let $x \randget S$ denote randomly sampling an element $x$ from a set $S$ Recall that $\langle g \rangle = \Gr$ denotes $g$ being a generator of the group $\Gr$ Definition of a pairing A pairing, also known as a bilinear map, is a function $e : \Gr_1 \times \Gr_2 \rightarrow \Gr_T$ between three groups $\Gr_1, \Gr_2$ and $\Gr_T$ of prime order $p$, with generators $g_1 = \langle \Gr_1 \rangle, g_2 = \langle \Gr_2 \rangle$ and $g_T = \langle \Gr_T \rangle$, respectively. When $\Gr_1 = \Gr_2$, the pairing is called symmetric. Otherwise, it is asymmetric. Most importantly, a pairing has two useful properties for cryptography: bilinearity and non-degeneracy.Hyperproofs: Faster Merkle proof aggregation without SNARKs2022-11-18T00:00:00+00:002022-11-18T00:00:00+00:00https://alinush.github.io//2022/11/18/Hyperproofs-faster-merkle-proof-aggregation-without-snark<p class="info"><strong>tl;dr:</strong> For now, see our Hyperproofs paper<sup id="fnref:SCPplus22" role="doc-noteref"><a href="#fn:SCPplus22" class="footnote" rel="footnote">1</a></sup>.</p>
<!--more-->
<p hidden="">$$
\def\Adv{\mathcal{A}}
\def\Badv{\mathcal{B}}
\def\vect#1{\mathbf{#1}}
$$</p>
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:SCPplus22" role="doc-endnote">
<p><strong>Hyperproofs: Aggregating and Maintaining Proofs in Vector Commitments</strong>, by Shravan Srinivasan and Alexander Chepurnoy and Charalampos Papamanthou and Alin Tomescu and Yupeng Zhang, <em>in 31st USENIX Security Symposium (USENIX Security 22)</em>, 2022, <a href="https://www.usenix.org/conference/usenixsecurity22/presentation/srinivasan">[URL]</a> <a href="#fnref:SCPplus22" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Alin Tomescutl;dr: For now, see our Hyperproofs paper[^SCPplus22].Are there cryptographic accumulators without trusted setup?2022-08-27T00:00:00+00:002022-08-27T00:00:00+00:00https://alinush.github.io//2022/08/27/Are-there-cryptographic-accumulators-without-trusted-setup<p class="info"><strong>tl;dr:</strong> Yes, there are: Merkle-based, RSA-or-class-group based and lattice-based ones.</p>
<!--more-->
<p hidden="">$$
\def\Adv{\mathcal{A}}
\def\Badv{\mathcal{B}}
\def\vect#1{\mathbf{#1}}
$$</p>
<hr />
<h2 id="rsa-accumulators-over-class-groups">RSA accumulators over class groups</h2>
<p>Practically, the only (somewhat-fast) accumulators <em>without</em> trusted setup (and constant-sized proofs) are RSA accumulators<sup id="fnref:Bd93" role="doc-noteref"><a href="#fn:Bd93" class="footnote" rel="footnote">1</a></sup> instantiated <em>with great care</em><sup id="fnref:BKSW20" role="doc-noteref"><a href="#fn:BKSW20" class="footnote" rel="footnote">2</a></sup> over class groups<sup id="fnref:Lipm12" role="doc-noteref"><a href="#fn:Lipm12" class="footnote" rel="footnote">3</a></sup>.</p>
<h2 id="merkle-based-accumulators">Merkle-based accumulators</h2>
<p>Theoretically 😄, if you relax your definition of “accumulators” by:</p>
<ol>
<li>Removing quasi-commutativity<sup id="fnref:Bd93:1" role="doc-noteref"><a href="#fn:Bd93" class="footnote" rel="footnote">1</a></sup>.</li>
<li>Allowing for logarithmically-sized proofs (instead of constant-sized proofs)</li>
</ol>
<p>…then, naturally you can use a Merkle prefix tree (a.k.a., a Merkle trie) to represent a set and obtain an accumulator.</p>
<p>Another approach is to either use (1) a binary search tree or (2) a tree with sorted leaves, where each internal node stores the minimum and the maximum element in its subtree<sup id="fnref:BLL00" role="doc-noteref"><a href="#fn:BLL00" class="footnote" rel="footnote">4</a></sup>.</p>
<p>Similarly, you can also use the rather beautiful Utreexo construction<sup id="fnref:Dryj19" role="doc-noteref"><a href="#fn:Dryj19" class="footnote" rel="footnote">5</a></sup>, which is also Merkle-based but does not support non-membership proofs.</p>
<h2 id="lattice-based-accumulators">Lattice-based accumulators</h2>
<p>Even more theoretically 😆, assuming you don’t care about performance at all, you might use a lattice-based accumulator<sup id="fnref:PSTY13" role="doc-noteref"><a href="#fn:PSTY13" class="footnote" rel="footnote">6</a></sup>,<sup id="fnref:JS14" role="doc-noteref"><a href="#fn:JS14" class="footnote" rel="footnote">7</a></sup>,<sup id="fnref:LLNW16" role="doc-noteref"><a href="#fn:LLNW16" class="footnote" rel="footnote">8</a></sup>,<sup id="fnref:LNWX17" role="doc-noteref"><a href="#fn:LNWX17" class="footnote" rel="footnote">9</a></sup>. Some of them do not need a trusted setup, like <sup id="fnref:PSTY13:1" role="doc-noteref"><a href="#fn:PSTY13" class="footnote" rel="footnote">6</a></sup>.</p>
<p>Even better, the recent lattice-based vector commitments by Peikert et al.<sup id="fnref:PPS21e" role="doc-noteref"><a href="#fn:PPS21e" class="footnote" rel="footnote">10</a></sup> can be turned into an accumulator. (Interestingly, I think accumulator proof sizes here could be made “almost” $O(\log_k{n})$-sized, for arbitrary $k$, if one used their lattice-based Verkle construction which, AFAICT, requires a trusted setup.)</p>
<h2 id="rsa-moduli-of-unknown-complete-factorization-ufos">RSA moduli of unknown complete factorization (UFOs)</h2>
<p>One last theoretical idea is to generate an RSA group with a modulus $N$ of unknown factorization using the <em>“RSA UFO”</em> technique by Sander<sup id="fnref:Sand99" role="doc-noteref"><a href="#fn:Sand99" class="footnote" rel="footnote">11</a></sup>. Unfortunatly, such $N$ are too large and kill performance.
Specifically, instead of the typical 2048-bit or 4096-bit, RSA UFO $N$’s are hundreds of thousands of bits (or larger?). Improving this would be a great avenue for future work.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:Bd93" role="doc-endnote">
<p><strong>One-Way Accumulators: A Decentralized Alternative to Digital Signatures</strong>, by Benaloh, Josh and de Mare, Michael, <em>in EUROCRYPT ‘93</em>, 1994 <a href="#fnref:Bd93" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:Bd93:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:BKSW20" role="doc-endnote">
<p><strong>A note on the low order assumption in class group of an imaginary quadratic number fields</strong>, by Karim Belabas and Thorsten Kleinjung and Antonio Sanso and Benjamin Wesolowski, <em>in Cryptology ePrint Archive, Report 2020/1310</em>, 2020, <a href="https://eprint.iacr.org/2020/1310">[URL]</a> <a href="#fnref:BKSW20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Lipm12" role="doc-endnote">
<p><strong>Secure Accumulators from Euclidean Rings without Trusted Setup</strong>, by Lipmaa, Helger, <em>in Applied Cryptography and Network Security</em>, 2012 <a href="#fnref:Lipm12" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:BLL00" role="doc-endnote">
<p><strong>Accountable certificate management using undeniable attestations</strong>, by Ahto Buldas and Peeter Laud and Helger Lipmaa, <em>in ACM CCS’00</em>, 2000, <a href="https://doi.org/10.1145%2F352600.352604">[URL]</a> <a href="#fnref:BLL00" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Dryj19" role="doc-endnote">
<p><strong>Utreexo: A dynamic hash-based accumulator optimized for the Bitcoin UTXO set</strong>, by Thaddeus Dryja, 2019, <a href="https://eprint.iacr.org/2019/611">[URL]</a> <a href="#fnref:Dryj19" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:PSTY13" role="doc-endnote">
<p><strong>Streaming Authenticated Data Structures</strong>, by Papamanthou, Charalampos and Shi, Elaine and Tamassia, Roberto and Yi, Ke, <em>in EUROCRYPT 2013</em>, 2013 <a href="#fnref:PSTY13" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:PSTY13:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:JS14" role="doc-endnote">
<p><strong>Compact Accumulator using Lattices</strong>, by Mahabir Prasad Jhanwar and Reihaneh Safavi-Naini, <em>in Cryptology ePrint Archive, Report 2014/1015</em>, 2014, <a href="https://eprint.iacr.org/2014/1015">[URL]</a> <a href="#fnref:JS14" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:LLNW16" role="doc-endnote">
<p><strong>Zero-Knowledge Arguments for Lattice-Based Accumulators: Logarithmic-Size Ring Signatures and Group Signatures Without Trapdoors</strong>, by Benoît Libert and San Ling and Khoa Nguyen and Huaxiong Wang, <em>in EUROCRYPT (2)</em>, 2016, <a href="https://www.iacr.org/archive/eurocrypt2016/96650306/96650306.pdf">[URL]</a> <a href="#fnref:LLNW16" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:LNWX17" role="doc-endnote">
<p><strong>Lattice-Based Group Signatures: Achieving Full Dynamicity with Ease</strong>, by San Ling and Khoa Nguyen and Huaxiong Wang and Yanhong Xu, <em>in Cryptology ePrint Archive, Report 2017/353</em>, 2017, <a href="https://eprint.iacr.org/2017/353">[URL]</a> <a href="#fnref:LNWX17" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:PPS21e" role="doc-endnote">
<p><strong>Vector and Functional Commitments from Lattices</strong>, by Chris Peikert and Zachary Pepin and Chad Sharp, <em>in Cryptology ePrint Archive, Report 2021/1254</em>, 2021, <a href="https://ia.cr/2021/1254">[URL]</a> <a href="#fnref:PPS21e" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Sand99" role="doc-endnote">
<p><strong>Efficient Accumulators without Trapdoor Extended Abstract</strong>, by Sander, Tomas, <em>in Information and Communication Security</em>, 1999 <a href="#fnref:Sand99" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Alin Tomescutl;dr: Yes, there are: Merkle-based, RSA-or-class-group based and lattice-based ones.Lagrange interpolation2022-07-28T10:38:00+00:002022-07-28T10:38:00+00:00https://alinush.github.io//2022/07/28/lagrange-interpolation<p>Recall from <a href="/2020/03/16/polynomials-for-crypto.html">our basics discussion</a> that a <strong>polynomial</strong> $\phi$ of <strong>degree</strong> $d$ is a vector of $d+1$ <strong>coefficients</strong>:</p>
<p>\begin{align}
\phi &= [\phi_0, \phi_1, \phi_2, \dots, \phi_d]
\end{align}</p>
<h2 id="how-to-compute-a-polynomials-coefficients-from-a-bunch-of-its-evaluations">How to compute a polynomial’s coefficients from a bunch of its evaluations</h2>
<p>Given $n$ pairs $(x_i, y_i)_{i\in[n]}$, one can compute or <em>interpolate</em> a degree $\le n-1$ polynomial $\phi(X)$ such that:
\(\phi(x_i)=y_i,\forall i\in[n]\)</p>
<p>Specifically, the <em>Lagrange interpolation</em> formula says that:
\begin{align}
\label{eq:lagrange-formula}
\phi(X) &= \sum_{i\in[n]} y_i \cdot \lagr_i(X),\ \text{where}\ \lagr_i(X) = \prod_{j\in[n],j\ne i} \frac{X-x_j}{x_i-x_j}
\end{align}</p>
<p>This formula is intimidating at first, but there’s a very simple intuition behind it.
The key idea is that $\lagr_i(X)$ is defined so that it has two properties:</p>
<ol>
<li>$\lagr_i(x_i) = 1,\forall i\in[n]$</li>
<li>$\lagr_i(x_j) = 0,\forall j \in [n]\setminus\{i\}$</li>
</ol>
<p>You can actually convince yourself that $\lagr_i(X)$ has these properties by plugging in $x_i$ and $x_j$ to see what happens.</p>
<p class="warning"><strong>Important:</strong> The $\lagr_i(X)$ polynomials are dependent on the set of $x_i$’s only (and thus on $n$)! Specifically each $\lagr_i(X)$ has degree $n-1$ and has a root at each $x_j$ when $j\ne i$!
In this sense, a better notation for them would be $\lagr_i^{[x_i, n]}(X)$ or $\lagr_i^{[n]}(X)$ to indicate this dependence.</p>
<h2 id="example-interpolating-a-polynomial-from-three-evaluations">Example: Interpolating a polynomial from three evaluations</h2>
<p>Consider the following example with $n=3$ pairs of points.
Then, by the Lagrange formula, we have:</p>
\[\phi(X) = y_1 \lagr_1(X) + y_2 \lagr_2(X) + y_3 \lagr_3(X)\]
<p>Next, by applying the two key properties of $\lagr_i(X)$ from above, you can easily check that $\phi(x_i) = y_i,\forall i\in[3]$:
\begin{align}
\phi(x_1) &= y_1 \lagr_1(x_1) + y_2 \lagr_2(x_1) + y_3 \lagr_3(x_1) = y_1 \cdot 1 + y_2 \cdot 0 + y_3 \cdot 0 = y_1\\<br />
\phi(x_2) &= y_1 \lagr_1(x_2) + y_2 \lagr_2(x_2) + y_3 \lagr_3(x_2) = y_1 \cdot 0 + y_2 \cdot 1 + y_3 \cdot 0 = y_2\\<br />
\phi(x_3) &= y_1 \lagr_1(x_3) + y_2 \lagr_2(x_3) + y_3 \lagr_3(x_3) = y_1 \cdot 0 + y_2 \cdot 0 + y_3 \cdot 1 = y_3
\end{align}</p>
<p>An <strong>important detail</strong> is that the degree of the interpolated $\phi(X)$ is $\le n-1$ and not necessarily exactly equal to $n-1$.
To see this, consider interpolating the polynomial $\phi(X)$ such that $\phi(i) = i$ for all $i\in [n]$.
In other words, $x_i = y_i = i$.</p>
<p>The inspired reader might notice that the polynomial $\phi(X) = X$ could satisfy our constraints.
But is this what the Lagrange interpolation will return?
After all, the interpolated $\phi(X)$ is a sum of degree $n-1$ polynomials $\lagr_i(X)$, so could it have degree 1?
Well, it turns out, yes, because things cancel out.
To see this, take a simple example, with $n=3$:
\begin{align}
\phi(X) &=\sum_{i\in [3]} i \cdot \lagr_i(X) = \sum_{i\in [3]} i \cdot \prod_{j\in[3]\setminus{i}} \frac{X - j}{i - j}\\<br />
&= 1\cdot \frac{X-2}{1-2}\frac{X-3}{1-3} + 2\cdot \frac{X-1}{2-1}\frac{X-3}{2-3} + 3\cdot\frac{X-1}{3-1}\frac{X-2}{3-2}\\<br />
&= \frac{X-2}{-1}\frac{X-3}{-2} + 2\cdot \frac{X-1}{1}\frac{X-3}{-1} + 3\cdot \frac{X-1}{2}\frac{X-2}{1}\\<br />
&= \frac{1}{2}(X-2)(X-3) - 2(X-1)(X-3) + \frac{3}{2}(X-1)(X-2)\\<br />
&= \frac{1}{2}[(X-2)(X-3) + 3(X-1)(X-2)] - 2(X-1)(X-3)\\<br />
&= \frac{1}{2}[(X-2)(4X-6)] - 2(X-1)(X-3)\\<br />
&= (X-2)(2X-3) - 2(X-1)(X-3)\\<br />
&= (2X^2 - 4X - 3X + 6) - 2(X^2 - 4X +3)\\<br />
&= (2X^2 - 7X + 6) - 2X^2 + 8X - 6\\<br />
&= X
\end{align}</p>
<h2 id="computational-overhead-of-lagrange-interpolation">Computational overhead of Lagrange interpolation</h2>
<p>If done naively, interpolating $\phi(X)$ using the Lagrange formula in Equation \ref{eq:lagrange-formula} will take $O(n^2)$ time.</p>
<p>However, there are known techniques for computing $\phi(X)$ in $O(n\log^2{n})$ time.
We described <strong>part of</strong> these techniques in a <a href="/2020/03/12/scalable-bls-threshold-signatures.html#our-quasilinear-time-bls-threshold-signature-aggregation">previous blog post</a>, but for the full techniques please refer to the <em>“Modern Computer Algebra”</em> book<sup id="fnref:vG13ModernCh10" role="doc-noteref"><a href="#fn:vG13ModernCh10" class="footnote" rel="footnote">1</a></sup>.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:vG13ModernCh10" role="doc-endnote">
<p><strong>Fast polynomial evaluation and interpolation</strong>, by von zur Gathen, Joachim and Gerhard, Jurgen, <em>in Modern Computer Algebra</em>, 2013 <a href="#fnref:vG13ModernCh10" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Alin TomescuRecall from our basics discussion that a polynomial $\phi$ of degree $d$ is a vector of $d+1$ coefficients: \begin{align} \phi &= [\phi_0, \phi_1, \phi_2, \dots, \phi_d] \end{align} How to compute a polynomial’s coefficients from a bunch of its evaluations Given $n$ pairs $(x_i, y_i)_{i\in[n]}$, one can compute or interpolate a degree $\le n-1$ polynomial $\phi(X)$ such that: \(\phi(x_i)=y_i,\forall i\in[n]\) Specifically, the Lagrange interpolation formula says that: \begin{align} \label{eq:lagrange-formula} \phi(X) &= \sum_{i\in[n]} y_i \cdot \lagr_i(X),\ \text{where}\ \lagr_i(X) = \prod_{j\in[n],j\ne i} \frac{X-x_j}{x_i-x_j} \end{align} This formula is intimidating at first, but there’s a very simple intuition behind it. The key idea is that $\lagr_i(X)$ is defined so that it has two properties: $\lagr_i(x_i) = 1,\forall i\in[n]$ $\lagr_i(x_j) = 0,\forall j \in [n]\setminus\{i\}$ You can actually convince yourself that $\lagr_i(X)$ has these properties by plugging in $x_i$ and $x_j$ to see what happens. Important: The $\lagr_i(X)$ polynomials are dependent on the set of $x_i$’s only (and thus on $n$)! Specifically each $\lagr_i(X)$ has degree $n-1$ and has a root at each $x_j$ when $j\ne i$! In this sense, a better notation for them would be $\lagr_i^{[x_i, n]}(X)$ or $\lagr_i^{[n]}(X)$ to indicate this dependence. Example: Interpolating a polynomial from three evaluations Consider the following example with $n=3$ pairs of points. Then, by the Lagrange formula, we have: \[\phi(X) = y_1 \lagr_1(X) + y_2 \lagr_2(X) + y_3 \lagr_3(X)\] Next, by applying the two key properties of $\lagr_i(X)$ from above, you can easily check that $\phi(x_i) = y_i,\forall i\in[3]$: \begin{align} \phi(x_1) &= y_1 \lagr_1(x_1) + y_2 \lagr_2(x_1) + y_3 \lagr_3(x_1) = y_1 \cdot 1 + y_2 \cdot 0 + y_3 \cdot 0 = y_1\\ \phi(x_2) &= y_1 \lagr_1(x_2) + y_2 \lagr_2(x_2) + y_3 \lagr_3(x_2) = y_1 \cdot 0 + y_2 \cdot 1 + y_3 \cdot 0 = y_2\\ \phi(x_3) &= y_1 \lagr_1(x_3) + y_2 \lagr_2(x_3) + y_3 \lagr_3(x_3) = y_1 \cdot 0 + y_2 \cdot 0 + y_3 \cdot 1 = y_3 \end{align} An important detail is that the degree of the interpolated $\phi(X)$ is $\le n-1$ and not necessarily exactly equal to $n-1$. To see this, consider interpolating the polynomial $\phi(X)$ such that $\phi(i) = i$ for all $i\in [n]$. In other words, $x_i = y_i = i$. The inspired reader might notice that the polynomial $\phi(X) = X$ could satisfy our constraints. But is this what the Lagrange interpolation will return? After all, the interpolated $\phi(X)$ is a sum of degree $n-1$ polynomials $\lagr_i(X)$, so could it have degree 1? Well, it turns out, yes, because things cancel out. To see this, take a simple example, with $n=3$: \begin{align} \phi(X) &=\sum_{i\in [3]} i \cdot \lagr_i(X) = \sum_{i\in [3]} i \cdot \prod_{j\in[3]\setminus{i}} \frac{X - j}{i - j}\\ &= 1\cdot \frac{X-2}{1-2}\frac{X-3}{1-3} + 2\cdot \frac{X-1}{2-1}\frac{X-3}{2-3} + 3\cdot\frac{X-1}{3-1}\frac{X-2}{3-2}\\ &= \frac{X-2}{-1}\frac{X-3}{-2} + 2\cdot \frac{X-1}{1}\frac{X-3}{-1} + 3\cdot \frac{X-1}{2}\frac{X-2}{1}\\ &= \frac{1}{2}(X-2)(X-3) - 2(X-1)(X-3) + \frac{3}{2}(X-1)(X-2)\\ &= \frac{1}{2}[(X-2)(X-3) + 3(X-1)(X-2)] - 2(X-1)(X-3)\\ &= \frac{1}{2}[(X-2)(4X-6)] - 2(X-1)(X-3)\\ &= (X-2)(2X-3) - 2(X-1)(X-3)\\ &= (2X^2 - 4X - 3X + 6) - 2(X^2 - 4X +3)\\ &= (2X^2 - 7X + 6) - 2X^2 + 8X - 6\\ &= X \end{align} Computational overhead of Lagrange interpolation If done naively, interpolating $\phi(X)$ using the Lagrange formula in Equation \ref{eq:lagrange-formula} will take $O(n^2)$ time. However, there are known techniques for computing $\phi(X)$ in $O(n\log^2{n})$ time. We described part of these techniques in a previous blog post, but for the full techniques please refer to the “Modern Computer Algebra” book1. Fast polynomial evaluation and interpolation, by von zur Gathen, Joachim and Gerhard, Jurgen, in Modern Computer Algebra, 2013 ↩Feist-Khovratovich technique for computing KZG proofs fast2021-06-17T00:00:00+00:002021-06-17T00:00:00+00:00https://alinush.github.io//2021/06/17/Feist-Khovratovich-technique-for-computing-KZG-proofs-fast<p class="info"><strong>tl;dr:</strong> Given a polynomial $f(X)$ of degree $m$, can we compute all $n$ <a href="/2020/05/06/kzg-polynomial-commitments.html">KZG</a> proofs for $f(\omega^k), k\in[0,n-1]$ in $O(n\log{n})$ time, where $\omega$ is a primitive $n$th root of unity?
Dankrad Feist and Dmitry Khovratovich<sup id="fnref:FK20" role="doc-noteref"><a href="#fn:FK20" class="footnote" rel="footnote">1</a></sup> give a resounding ‘yes!’</p>
<!--more-->
<p hidden="">$$
\def\Adv{\mathcal{A}}
\def\Badv{\mathcal{B}}
\def\vect#1{\mathbf{#1}}
%\definecolor{myBlueColor}{HTML}{268BD2}
%\definecolor{myPinkColor}{HTML}{D33682}
%\definecolor{myGreenColor}{HTML}{859900}
%\def\mygreen#1{\color{myGreenColor}{#1}}
%\def\mygreen#1{\color{green}{#1}}
%\newcommand{\myblue}[1]{\textcolor{myBlueColor}{#1}}
%\newcommand{\mypink}[1]{\textcolor{myPinkColor}{#1}}
$$</p>
<h2 id="preliminaries">Preliminaries</h2>
<p>First, read:</p>
<ul>
<li><a href="/2020/03/16/polynomials-for-crypto.html">Basics of polynomials</a></li>
<li>The <a href="https://en.wikipedia.org/wiki/Discrete_Fourier_transform_(general)">Discrete Fourier Transform (DFT)</a> for $n$th roots of unity</li>
<li><a href="/2020/05/06/kzg-polynomial-commitments.html">KZG polynomial commitments</a></li>
</ul>
<p>Notation:</p>
<ul>
<li>$g$ generates a “bilinear” group $\Gr$ of prime order $p$, endowed with a <em>bilinear map</em> or <em>pairing</em> $e : \Gr\times\Gr\rightarrow\Gr_T$
<ul>
<li>we use multiplicative notation here: i.e., $g^a$ denotes composing $g$ with itself $a$ times</li>
</ul>
</li>
<li>there exists a primitive $n$th root of unity $\omega$, where $n$ is a power of two</li>
<li>we have $q$-SDH public parameters $\left(g, g^\tau,g^{\tau^2},\dots,g^{\tau^q}\right)$, for $q \le n$</li>
<li>we work with polynomials in $\Zp[X]$ of degree $\le n$</li>
</ul>
<h2 id="related-work">Related work</h2>
<p>Other works that give related techniques to compute proofs fast in KZG-like polynomial commitments are:</p>
<ul>
<li><strong>Authenticated multipoint evaluation trees (AMTs)</strong> (see paper<sup id="fnref:TCZplus20" role="doc-noteref"><a href="#fn:TCZplus20" class="footnote" rel="footnote">2</a></sup> and <a href="/2020/03/12/towards-scalable-vss-and-dkg.html">blogpost</a>)</li>
<li><strong>“How to compute all Pointproofs”</strong> (see paper<sup id="fnref:Tome20How" role="doc-noteref"><a href="#fn:Tome20How" class="footnote" rel="footnote">3</a></sup>), much inspired by Feist and Khovratovich’s technique explained in this blogpost</li>
</ul>
<h2 id="refresher-computing-n-different-kzg-proofs">Refresher: Computing $n$ different KZG proofs</h2>
<p>Let $f(X)$ be a polynomial with coefficients $f_i$:</p>
<p>\begin{align*}
f(X) &= f_m X^m + f_{m-1} X^{m-1} + \cdots f_1 X + f_0\\<br />
&= \sum_{i\in[0,m]} f_i X^i
\end{align*}</p>
<p><a href="/2020/05/06/kzg-polynomial-commitments.html">Recall that</a> a KZG <strong>evaluation proof</strong> $\pi_i$ for $f(\omega^i)$ is a KZG commitment to a <strong>quotient polynomial</strong> $Q_i(X) = \frac{f(X) - f(\omega^i)}{X-\omega^i}$:
\begin{align*}
\pi_i = g^{Q_i(\tau)} = g^{\frac{f(\tau) - f(\omega^i)}{\tau-\omega^i}}
\end{align*}</p>
<p>Computing such a proof takes $O(m)$ time!</p>
<p><strong>But what if we want to compute all</strong> $\pi_i, i\in[0, n)$?</p>
<p>If done naively, this would take $O(nm)$ time, which is <strong>too expensive!</strong></p>
<p class="warning"><strong>Lower bound?</strong>
Is this $O(nm)$ time complexity inherent? After all, to compute $\pi_i$ don’t we first need to compute $Q_i(X)$, which takes $O(m)$ time? As you’ll see below, the answer is <strong>“no!”</strong></p>
<p>Fortunately, Feist and Khovratovich observe that the $Q_i$’s are algebraically-related and so are their $\pi_i$ KZG commitments!
As a result, they observe that computing all $\pi_i$’s does <strong>not</strong> require computing all $Q_i$’s.</p>
<p>Below, we explain how their faster, $O(n\log{n})$-time technique works!</p>
<p>An important <strong>caveat</strong> is that their technique relies on the evaluation points being $\omega^0,\dots,\omega^{n-1}$.</p>
<h2 id="quotient-polynomials-and-their-coefficients">Quotient polynomials and their coefficients</h2>
<p>To understand how the $\pi_i$’s relate to one other, let us look at the coefficients of $Q_i(X)$.</p>
<p>We can show that when dividing $f$ (of degree $m$) by $(X-\omega^i)$ we obtain a <em>quotient polynomial</em> with coefficients $t_0, t_1, \dots, t_{m-1}$ such that:
\begin{align}
\label{eq:div-coeffs-1}
t_{m-1} &= f_m\\<br />
%t_{m-2} &= f_{m-1} + \omega^i \cdot f_m\\<br />
%t_{m-3} &= f_{m-2} + \omega^i \cdot t_{m-2}\\<br />
% &= f_{m-2} + \omega^i \cdot (f_{m-1} + \omega^i \cdot f_m)\\<br />
% &= f_{m-2} + \omega^i \cdot f_{m-1} + \omega^{2i} \cdot f_m\\<br />
%t_{m-4} &= f_{m-3} + \omega^i \cdot t_{m-3}\\<br />
% &= f_{m-3} + \omega^i \cdot (f_{m-2} + \omega^i \cdot f_{m-1} + \omega^{2i} \cdot f_m)\\<br />
% &= f_{m-3} + \omega^i \cdot f_{m-2} + \omega^{2i} \cdot f_{m-1} + \omega^{3i} \cdot f_m\\\
% & \vdots\\<br />
\label{eq:div-coeffs-2}
t_{j} &= f_{j+1} + \omega^i \cdot t_{j+1}, \forall j \in [0, m-1)
% & \vdots\\<br />
% t_0 &= f_1 + \omega^i \cdot f_2 + \omega^{2i} f_3 + \dots + \omega^{m-1} f_m
\end{align}
Note that the $t_i$’s are a function of $f_m, f_{m-1},\dots, f_1$, but not of $f_0$!</p>
<details class="info">
<summary>
<b style="color: #f5222d">Proof:</b> One could prove by induction that the coefficients above are correct.
However, it's easiest to take an example and convince yourself, as shown below for $m=4$.
(Click to expand.)
</summary>
<div style="margin-left: .3em; border-left: .15em solid #f5222d; padding-left: .5em;"><p>
Indeed, the quotient obtained when dividing $f(X) = f_3 X^3 + f_2 X^2 + \dots + f_0$ by $X-\omega^i$ exactly matches Equations \ref{eq:div-coeffs-1} and \ref{eq:div-coeffs-2} above:
<!-- WARNING: No support for cline in MathJax, so that's why this is a .png -->
<div align="center">
<a href="/pictures/fk-division-example.png">
<img style="width:95%" src="/pictures/fk-division-example.png" />
</a>
</div>
Specifically, the quotient's coefficients, as expected, are:
\begin{align}
t_2 &= \color{green}{f_3}\\
t_1 &= f_2 + \omega^i t_2\\
&= \color{blue}{f_2 + \omega^i f_3}\\
t_0 &= f_1 + \omega^i t_1 = f_1 + \omega^i \cdot (f_2 + \omega^i f_3)\\
&= \color{pink}{f_1 + \omega^i f_2 + \omega^{2i} f_3}
\end{align}
</p></div>
</details>
<h2 id="the-relationship-between-kzg-proofs">The relationship between KZG proofs</h2>
<p>Next, let us expand Equations \ref{eq:div-coeffs-1} and \ref{eq:div-coeffs-2} above and get a better sense of the relationship between KZG quotient polynomials:
\begin{align}
\color{green}{t_{m-1}} &= \underline{f_m}\\<br />
\color{blue}{t_{m-2}} &= f_{m-1} + \omega^i \cdot \color{green}{t_{m-1}} =\nonumber\\<br />
&= \underline{f_{m-1} + \omega^i \cdot f_m}\\<br />
\color{red}{t_{m-3}} &= f_{m-2} + \omega^i \cdot \color{blue}{t_{m-2}}\nonumber\\<br />
&= f_{m-2} + \omega^i \cdot (f_{m-1} + \omega^i \cdot f_m)\nonumber\\<br />
&= \underline{f_{m-2} + \omega^i \cdot f_{m-1} + \omega^{2i} \cdot f_m}\\<br />
t_{m-4} &= f_{m-3} + \omega^i \cdot \color{red}{t_{m-3}}\nonumber\\<br />
&= f_{m-3} + \omega^i \cdot (f_{m-2} + \omega^i \cdot f_{m-1} + \omega^{2i} \cdot f_m)\nonumber\\<br />
&= \underline{f_{m-3} + \omega^i \cdot f_{m-2} + \omega^{2i} \cdot f_{m-1} + \omega^{3i} \cdot f_m}\\\
&\hspace{.55em}\vdots\nonumber\\<br />
% t_{j} &= f_{j+1} + \omega^i \cdot t_{j+1}, \forall j \in [0, m-1)\\<br />
% & \vdots\\<br />
t_1 &= \underline{f_2 + \omega^i \cdot f_3 + \omega^{2i} \cdot f_4 + \dots + \omega^{(m-2)i} \cdot f_m}\\<br />
t_0 &= \underline{f_1 + \omega^i \cdot f_2 + \omega^{2i} \cdot f_3 + \dots + \omega^{(m-1)i} \cdot f_m}
\end{align}
As you can see above, the quotient polynomial $Q_i(X) = \sum_{j=0}^{m-1} t_j X^j$ obtained when dividing $f(X)$ by $X-\omega^i$ is:
\begin{align}
Q_i(X) &= f_m \cdot X^{m-1} + {}\\<br />
&+ \left(f_{m-1} + \omega^i \cdot f_m\right) \cdot X^{m-2} + {}\nonumber\\<br />
&+ \left(f_{m-2} + \omega^i \cdot f_{m-1} + \omega^{2i} \cdot f_m\right) \cdot X^{m-3} + {}\nonumber\\<br />
&+ \left(f_{m-3} + \omega^i \cdot f_{m-2} + \omega^{2i} \cdot f_{m-1} + \omega^{3i} \cdot f_m\right) \cdot X^{m-4} + {}\nonumber\\\
&+ \dots + {}\nonumber\\\
&+ \left(f_2 + \omega^i \cdot f_3 + \omega^{2i} \cdot f_4 + \dots + \omega^{(m-2)i} \cdot f_m\right) \cdot X + {}\nonumber\\<br />
&+ \left(f_1 + \omega^i \cdot f_2 + \omega^{2i} \cdot f_3 + \dots + \omega^{(m-1)i} \cdot f_m\right)\nonumber
\end{align}
Factoring out the roots of unity, we can rearrange this as follows:
\begin{align}
\label{eq:HX}
Q_i(X) &= \left(f_m X^{m-1} + f_{m-1} X^{m-2} + \dots + f_1\right) (\omega^i)^0 + {}\\<br />
&+ \left(f_m X^{m-2} + f_{m-1} X^{m-3} + \dots + f_2\right) (\omega^i)^1 + {}\nonumber\\<br />
&+ \left(f_m X^{m-3} + f_{m-1} X^{m-4} + \dots + f_3\right) (\omega^i)^2 + {}\nonumber\\<br />
&+ \dots + {}\nonumber\\<br />
&+ \left(f_m X + f_{m-1}\right) (\omega^i)^{m-2} + {}\nonumber\\<br />
&+ \left(f_m \right) (\omega^i)^{m-1}\nonumber
\end{align}
Baptising the polynomials above as $H_j(X)$, we can rewrite as:
\begin{align}
Q_i(X) &\bydef H_1(X) (\omega^i)^0 + {}\\<br />
&+ H_2(X) (\omega^i)^1 + {}\nonumber\\<br />
&+ \dots + {}\nonumber\\<br />
&+ H_m(X) (\omega^i)^{m-1}\nonumber\\<br />
\end{align}
More succinctly, the quotient polynomial is:
\begin{align}
\label{eq:Qi-poly}
Q_i(X) &= \sum_{k=0}^{m-1} H_{j+1}(X) \cdot (\omega^i)^k
\end{align}</p>
<p class="error"><strong>Note:</strong> At this point, it is not helpful to write down a closed form formula for $H_j(X)$, but we’ll return to it later.</p>
<p>Next, let:
\begin{align}
\label{eq:hj}
h_j = g^{H_j(\tau)},\forall j\in[m]
\end{align}
…denote a KZG commitment to $H_j(X)$.
(We are ignoring for now the actual closed-form formula for the $H_j$’s.)</p>
<p>Recall that
<!--\begin{align}-->
\(\pi_i=g^{Q_i(\tau)}\)
<!--\end{align}-->
denotes a KZG proof for $\omega^i$.</p>
<p>Therefore, applying Equation \ref{eq:Qi-poly} to $\pi_i$’s expression, we get:
\begin{align}
\label{eq:pi-dft-like}
\pi_i = \prod_{j=0}^{m-1} \left(h_{j+1}\right)^{(\omega^i)^j}, \forall i\in[0,n)
\end{align}</p>
<p>But a close look at Equation \ref{eq:pi-dft-like} reveals it is actually a <strong>Discrete Fourier Transform (DFT)</strong> on the $h_j$’s!
Specifically, we can rewrite it as:
\begin{align}
\label{eq:pi-dft}
[ \pi_0, \pi_1, \dots, \pi_{n-1} ] = \mathsf{DFT}_{\Gr}(h_1, h_2, \dots, h_m, h_{m+1},\dots, h_n)
\end{align}
Here, the extra $h_{m+1},\dots,h_n$ (if any) are just commitments to the zero polynomials: i.e., they are the identity element in $\Gr$.
(Also, $\mathsf{DFT}_{\Gr}$ is a DFT on group elements via exponentiations, rather than on field elements via multiplication.)</p>
<p class="info"><strong>Time complexity:</strong> Ignoring the time to compute the $h_j$ commitments, which we have not discussed yet, note that the DFT above would only take $O(n\log{n})$ time!</p>
<p>This (almost) summarizes the <strong>Feist-Khovratovich (FK)</strong> technique!</p>
<p><strong>The key idea?</strong> KZG quotient polynomial commitments are actually related, if the evaluation points are roots of unity.
Specifically, these commitments are the output of a single DFT as per Equation \ref{eq:pi-dft}, which can be computed in quasilinear time!</p>
<p>However, <strong>one key challenge remains</strong>, which we address next: computing the $h_j$ commitments.</p>
<h2 id="computing-the-h_j--gh_jtau-commitments">Computing the $h_j = g^{H_j(\tau)}$ commitments</h2>
<p>To see how the $h_j$’s can be computed fast too, let’s rewrite them from Equation \ref{eq:HX}.
\begin{align}
H_1(X) &= f_m X^{m-1} + f_{m-1} X^{m-2} + \dots + f_1\\<br />
H_2(X) &= f_m X^{m-2} + f_{m-1} X^{m-3} + \dots + f_2\\<br />
H_3(X) &= f_m X^{m-3} + f_{m-1} X^{m-4} + \dots + f_3\\<br />
&\vdots\\<br />
H_m(X) &= f_m X + f_{m-1}\\<br />
H_{m-1}(X) &= f_m
\end{align}
<strong>Key observation:</strong> We can express the $H_j(X)$ polynomials as a <a href="/2020/03/19/multiplying-a-vector-by-a-toeplitz-matrix.html">Toeplitz matrix product</a> between a matrix $\mathbf{F}$ (of $f(X)$’s coefficients) and a column vector $V(X)$ (of the indeterminate variable $X$):
\begin{align}
\begin{bmatrix}
H_1(X)\\<br />
H_2(X)\\<br />
H_3(X)\\<br />
\vdots\\<br />
H_m(X)\\<br />
H_{m-1}(X)\\<br />
\end{bmatrix}
&=
\begin{bmatrix}
f_m & f_{m-1} & f_{m-2} & f_{m-3} & \dots & f_2 & f_1\\<br />
0 & f_m & f_{m-1} & f_{m-2} & \dots & f_3 & f_2\\<br />
0 & 0 & f_m & f_{m-1} & \dots & f_4 & f_3\\<br />
\vdots & & & \ddots & & & \vdots\\<br />
0 & 0 & 0 & 0 & \dots & f_m & f_{m-1}\\<br />
0 & 0 & 0 & 0 & \dots & 0 & f_m
\end{bmatrix}
\cdot
\begin{bmatrix}
X^{m-1}\\<br />
X^{m-2}\\<br />
X^{m-3}\\<br />
\vdots\\<br />
X\\<br />
1
\end{bmatrix}
\\<br />
&\bydef
\mathbf{F} \cdot V(X)
\end{align}
Therefore, the commitments $h_j$ to the $H_j(X)$’s can also be expressed as a Toeplitz matrix product, where “multiplication” is replaced with “exponentation” and the column vector $V(X)$ is replaced by $V(\tau)$:
\begin{align}
\begin{bmatrix}
h_1\\<br />
h_2\\<br />
h_3\\<br />
\vdots\\<br />
h_m\\<br />
h_{m-1}\\<br />
\end{bmatrix}
&=
\begin{bmatrix}
f_m & f_{m-1} & f_{m-2} & f_{m-3} & \dots & f_2 & f_1\\<br />
0 & f_m & f_{m-1} & f_{m-2} & \dots & f_3 & f_2\\<br />
0 & 0 & f_m & f_{m-1} & \dots & f_4 & f_3\\<br />
\vdots & & & \ddots & & & \vdots\\<br />
0 & 0 & 0 & 0 & \dots & f_m & f_{m-1}\\<br />
0 & 0 & 0 & 0 & \dots & 0 & f_m
\end{bmatrix}
\cdot
\begin{bmatrix}
\tau^{m-1}\\<br />
\tau^{m-2}\\<br />
\tau^{m-3}\\<br />
\vdots\\<br />
\tau\\<br />
1
\end{bmatrix}
\\<br />
&\bydef
\mathbf{F} \cdot V(\tau)
\end{align}
Fortunately, it is well known that such a matrix product can be computed in $O(m\log{m})$ time (incidentally, also via DFTs).
If you are curious, in a <a href="/2020/03/19/multiplying-a-vector-by-a-toeplitz-matrix.html">previous blogpost</a>, as well as in a short paper<sup id="fnref:Tome20How:1" role="doc-noteref"><a href="#fn:Tome20How" class="footnote" rel="footnote">3</a></sup>, we explain in detail how this works.</p>
<h2 id="conclusion">Conclusion</h2>
<p><strong>We are all done!</strong>
To summarize, to compute all proofs $\pi_i$ for $f(\omega^i)$, the <em>Feist-Khovratovich (FK)</em> technique<sup id="fnref:FK20:1" role="doc-noteref"><a href="#fn:FK20" class="footnote" rel="footnote">1</a></sup> proceeds as follows:</p>
<ol>
<li>Computes all $h_j$’s from Equation \ref{eq:hj} in $O(m\log{m})$ time via a Toeplitz matrix product</li>
<li>Computes all $\pi_i$’s in $O(n\log{n})$ time via a DFT on the $h_j$’s, as per Equation \ref{eq:pi-dft}</li>
</ol>
<p>A few things that we could still talk about, but we are out of time:</p>
<ul>
<li>Implementing this efficiently (see one attempt <a href="https://github.com/alinush/libpolycrypto/blob/fk/libpolycrypto/bench/BenchFk.cpp">here</a>)</li>
<li>Optimizing part of the implementation (see Dankrad’s observation’s in <a href="https://twitter.com/alinush407/status/1360228894851305475">this tweet</a>)</li>
<li>Other techniques for computing proofs on multiple polynomials from the FK paper<sup id="fnref:FK20:2" role="doc-noteref"><a href="#fn:FK20" class="footnote" rel="footnote">1</a></sup></li>
<li>Decreasing KZG verifier time when opening multiple polynomials $(f_i)_{i\in[t]}$ at the same point $x=z^t$, also via the power of DFTs<sup id="fnref:GW21" role="doc-noteref"><a href="#fn:GW21" class="footnote" rel="footnote">4</a></sup></li>
</ul>
<!--
General formula for $H_j$'s:
\begin{align}
H_j(X) \bydef \sum_{k=j}^m f_{k} X^{k-j}, \forall j\in[1,m]
\end{align}
-->
<!--
Sanity check:
H_1(X) = f_1 X^{1-1} + f_2 X^{2-1} + f_3 X^{3-1} + \dots + f_m X^{m-1}
H_2(X) = f_2 X^{2-2} + f_3 X^{3-2} + f_4 X^{4-2} + \dots + f_m X^{m-2}
\vdots
H_j(X) = \prod_{k = j}^m f_k X^{k-j}
\vdots
H_{m-1}(X) = f_{m-1} X^{(m-1) - (m-1)} f_m X^{m-(m-1)} = f_{m-1} + f_m X
H_m(X) = f_m X^{m-m} = f_m
-->
<hr />
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:FK20" role="doc-endnote">
<p><strong>Fast amortized Kate proofs</strong>, by Dankrad Feist and Dmitry Khovratovich, 2020, <a href="https://github.com/khovratovich/Kate">[URL]</a> <a href="#fnref:FK20" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:FK20:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a> <a href="#fnref:FK20:2" class="reversefootnote" role="doc-backlink">↩<sup>3</sup></a></p>
</li>
<li id="fn:TCZplus20" role="doc-endnote">
<p><strong>Towards Scalable Threshold Cryptosystems</strong>, by Alin Tomescu and Robert Chen and Yiming Zheng and Ittai Abraham and Benny Pinkas and Guy Golan Gueta and Srinivas Devadas, <em>in IEEE S\&P’20</em>, 2020 <a href="#fnref:TCZplus20" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:Tome20How" role="doc-endnote">
<p><strong>How to compute all Pointproofs</strong>, by Alin Tomescu, <em>in Cryptology ePrint Archive, Report 2020/1516</em>, 2020, <a href="https://eprint.iacr.org/2020/1516">[URL]</a> <a href="#fnref:Tome20How" class="reversefootnote" role="doc-backlink">↩</a> <a href="#fnref:Tome20How:1" class="reversefootnote" role="doc-backlink">↩<sup>2</sup></a></p>
</li>
<li id="fn:GW21" role="doc-endnote">
<p><strong>fflonk: a Fast-Fourier inspired verifier efficient version of PlonK</strong>, by Ariel Gabizon and Zachary J. Williamson, <em>in Cryptology ePrint Archive, Report 2021/1167</em>, 2021, <a href="https://ia.cr/2021/1167">[URL]</a> <a href="#fnref:GW21" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Alin Tomescutl;dr: Given a polynomial $f(X)$ of degree $m$, can we compute all $n$ KZG proofs for $f(\omega^k), k\in[0,n-1]$ in $O(n\log{n})$ time, where $\omega$ is a primitive $n$th root of unity? Dankrad Feist and Dmitry Khovratovich[^FK20] give a resounding ‘yes!’Basic number theory2021-04-15T00:00:00+00:002021-04-15T00:00:00+00:00https://alinush.github.io//2021/04/15/basic-number-theory<!-- TODO: totient; gcd's -->
<h2 id="multiplicative-inverses-modulo-m">Multiplicative inverses modulo $m$</h2>
<p>The multiplicative group of integers modulo $m$ is defined as:
\begin{align}
\Z_m^* = \{a\ |\ \gcd(a,m) = 1\}
\end{align}
But why?
This is because Euler’s theorem says that:
\begin{align}
\gcd(a,m) = 1\Rightarrow a^{\phi(m)} = 1
\end{align}
This in turn, implies that every element in $\Z_m^*$ has an inverse, since:
\begin{align}
a\cdot a^{\phi(m) - 1} &= 1
\end{align}
Thus, for a prime $p$, all elements in $\Z_p^* = \{1,2,\dots, p-1\}$ have inverses.
Specifically, the inverse of $a \in \Z_p^*$ is $a^{p-2}$.</p>Alin TomescuMultiplicative inverses modulo $m$ The multiplicative group of integers modulo $m$ is defined as: \begin{align} \Z_m^* = \{a\ |\ \gcd(a,m) = 1\} \end{align} But why? This is because Euler’s theorem says that: \begin{align} \gcd(a,m) = 1\Rightarrow a^{\phi(m)} = 1 \end{align} This in turn, implies that every element in $\Z_m^*$ has an inverse, since: \begin{align} a\cdot a^{\phi(m) - 1} &= 1 \end{align} Thus, for a prime $p$, all elements in $\Z_p^* = \{1,2,\dots, p-1\}$ have inverses. Specifically, the inverse of $a \in \Z_p^*$ is $a^{p-2}$.What is a Merkle tree?2021-02-25T00:00:00+00:002021-02-25T00:00:00+00:00https://alinush.github.io//2021/02/25/what-is-a-merkle-tree<p class="info"><strong>tl;dr:</strong> In this post, we demystify Merkle trees for beginners.
We give simple, illustrated explanations, we detail applications they are used in and reason about their security formally.</p>
<!-- more -->
<p>For more details, see this full post on <a href="https://decentralizedthoughts.github.io/2020-12-22-what-is-a-merkle-tree/">Decentralized Thoughts</a>.</p>Alin Tomescutl;dr: In this post, we demystify Merkle trees for beginners. We give simple, illustrated explanations, we detail applications they are used in and reason about their security formally. For more details, see this full post on Decentralized Thoughts.