Nomadic Labshttps://research-development.nomadic-labs.com/2024-03-08T16:00:00+01:0010 Second Blocks: A Faster Tezos, Fully Decentralized2024-03-08T16:00:00+01:002024-03-08T16:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2024-03-08:/10-second-blocktime.html<p>Lower latency, faster finality, no compromise on decentralization.</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: Based on meticulous testing, Nomadic Labs and Trilitech propose reducing Tezos’ Layer 1 block time from 15 to 10 seconds. This blog post walks you through the test setup and results, including a hardware recommendation for bakers.</strong></p>
<p><em>Block time</em> is an important and highly visible parameter for a blockchain. </p>
<p>It determines how often incoming transactions are added to the chain, and on Tezos it impacts the user experience in two noticeable ways:<em> latency</em> and <em>finality</em>.</p>
<p><em>Latency</em> is how quickly a new transaction is first applied by the network. The shorter the block time, the shorter the wait before moving on with what you were doing, and the smoother the user experience.</p>
<p><em>Finality</em> is the time it takes for a transaction to be considered irreversible. The shorter the time to finality, the smaller the window for malicious actors to attempt double-spending or other fraudulent activities. Faster finality makes a blockchain more reliable and useful for mission critical and high-value transactions.</p>
<p>Tezos’ <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">‘Tenderbake’</a> consensus algorithm achieves finality after two blocks, and the current block time of 15 seconds hence equals a time to finality of around 30 seconds. In comparison, the time to finality on Ethereum is <a href="https://ethereum.org/en/roadmap/single-slot-finality/">~15 minutes</a> (64+ blocks), and on Solana it’s <a href="https://docs.solanalabs.com/consensus/commitments">12 seconds</a> (32 blocks).</p>
<h2 id="10-second-blocks-secure-stable-decentralized"><a class="toclink" href="#10-second-blocks-secure-stable-decentralized">10 second blocks: secure, stable, decentralized</a></h2>
<p>Tezos launched with a Layer 1 block time of 60 seconds in 2018, and developer teams have since worked on optimizing the network to enable shorter block times. In 2021 block time <a href="https://research-development.nomadic-labs.com/announcing-granada.html">was reduced</a> to 30 seconds, and in 2023 it was <a href="https://research-development.nomadic-labs.com/mumbai-preview.html#pipelining-is-fully-deployed-giving-us-15-second-block-times">further reduced</a> to the current 15 seconds. </p>
<p>In an upcoming “P” protocol proposal, we include a reduction to 10 seconds, enabling a time to finality of 20 seconds. </p>
<p>Why not just step on the pedal and lower block time to, say, 1 second? </p>
<p>As with most parameters – block size being another example – it’s not quite that simple. Each blockchain network works differently, but they share common challenges in that they must balance speed with other factors, notably:</p>
<ul>
<li><strong>Security.</strong> Transactions are sufficiently verified before being accepted by the network.</li>
<li><strong>Stability.</strong> The blockchain runs reliably, with outages close to non-existent. Also called <em>liveness</em>.</li>
<li><strong>Decentralization.</strong> Broad distribution of validators prevents colluding actors from selectively censoring transactions. </li>
</ul>
<p>While most blockchains more or less align on the first two, decentralization is often treated differently by various blockchains with diverging philosophies and assumptions.</p>
<p>To achieve higher speeds, some blockchains define a limited set of validators that secure the network. Others have high requirements in terms of hardware, internet connection, and minimum stake, effectively creating an economic barrier.</p>
<p>The Tezos ecosystem prioritizes open participation and a low barrier of entry for bakers (validators). The goal is to have bakers of different geographical regions and economic abilities participate in securing the network, which encourages decentralization and makes the network more inclusive.</p>
<h2 id="recommended-hardware"><a class="toclink" href="#recommended-hardware">Recommended hardware</a></h2>
<p>With recent optimizations of the network, we are confident that a block time of 10 seconds will not harm security, stability or decentralization.</p>
<p>The conclusion is based on thorough testing, performed with strict performance requirements, high load on the network (full blocks), and with added delays simulating Ledger Nano S signing and unstable network conditions. </p>
<p>In fact, the test was performed successfully with a block time of 8 seconds, and 10 seconds was chosen to provide extra safety margin. For a deep dive into our testing methodology and the optimizations that made the reduction possible, see sections below.</p>
<p>Based on our results, we recommend the following minimum hardware specifications, which are viable for all and performant enough to bake at the target block time. </p>
<ul>
<li>3 <span class="caps">CPU</span> cores</li>
<li><span class="caps">8GB</span> of <span class="caps">RAM</span></li>
<li><span class="caps">100GB</span> <span class="caps">SSD</span> storage (or similar I/O performance)</li>
<li>A low latency reliable internet connection </li>
</ul>
<p>As the work to reduce latency and time to finality on the Tezos network continues, we expect further reductions in Layer 1 block time, but low hardware requirements remains a goal.</p>
<p>And now, let’s dive into the details behind the reduction to 10 seconds. Heads up, it’s going to get technical.</p>
<h2 id="the-road-to-10-seconds"><a class="toclink" href="#the-road-to-10-seconds">The road to 10 seconds</a></h2>
<p>We start with a brief history of Tezos network optimizations and block time reductions.</p>
<p>In 2021, a reduction from 60 to 30 seconds was made possible with tweaks to the consensus algorithm (<a href="https://research-development.nomadic-labs.com/faster-finality-with-emmy.html">Emmy*</a>) in protocol <a href="https://tezos.gitlab.io/protocols/010_granada.html#emmy">Granada</a>. </p>
<p>In 2022, protocol <a href="https://research-development.nomadic-labs.com/ithaca-is-live.html">Ithaca</a> replaced Emmy* with <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">Tenderbake</a>, a brand new consensus algorithm with deterministic finality, which opened the door to further optimizations. Ithaca also introduced a ‘light check’ validation of manager operations (such as transfers and contract calls) allowing nodes to propagate blocks faster through the network. It also introduced mempool optimizations. The light check approach was later improved in protocol <a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html#first-steps-towards-pipelined-block-validation">Kathmandu</a>, as part of the <a href="https://gitlab.com/tezos/tezos/-/milestones/23#tab-issues">pipelining project</a>. </p>
<p>In 2023, protocol <a href="https://research-development.nomadic-labs.com/lima-preview.html#more-pipelining-for-a-faster-blockchain">Lima</a> extended the light check concept to other types of operations (consensus, voting, etc.). Building on these advancements, the block time was reduced to 15 seconds in the <a href="https://research-development.nomadic-labs.com/mumbai-preview.html#pipelining-is-fully-deployed-giving-us-15-second-block-times">Mumbai</a> protocol. </p>
<p>More recently, in the <a href="https://research-development.nomadic-labs.com/nairobi-announcement.html#fine-tuned-validation-pipelining-for-faster-consensus">Nairobi</a> protocol, consensus operations propagation was again improved to further accelerate the consensus process.</p>
<p>At this point, identifying the obstacles for further block time reduction was not straightforward, and we deemed it necessary to improve our methodology with reproducible large-scale experiments.</p>
<p>Below, we cover the setup, the results, identified obstacles and improvements, and next steps.</p>
<h2 id="methodology"><a class="toclink" href="#methodology">Methodology</a></h2>
<p>The purpose of our experiments is to examine whether a given block time is “safe”, and if not, identify bottlenecks. We define a safe block time as one where network performance lives up to these criteria:</p>
<ul>
<li>at least 99% of blocks are baked at <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html#levels-are-composed-of-rounds">round</a> 0</li>
<li>no blocks are baked with a round higher than 1</li>
</ul>
<p><strong>Framework</strong></p>
<p>To get meaningful data about the consequences of a reduced block time in our simulations, we need to mimic Tezos Mainnet as closely as possible and load the chain with heavy traffic. </p>
<p>For this we have developed the framework <em>Tzimulator</em>, which is capable of executing large-scale experiments within a controlled environment. </p>
<p>The tool uses real-world data in the form of historical operations from Tezos Mainnet. It takes a snapshot of the chain as a starting point, and executes the subsequent two weeks of operations on top of it.</p>
<ul>
<li>A cluster of Tezos nodes is started</li>
<li>The snapshot from Tezos Mainnet is imported</li>
<li>Using the historical operations, an <em>injector</em> node keeps a high level of proposed operations in the mempool, ensuring full blocks (at the gas limit) throughout the test</li>
<li>The test is executed for a specified number of operations or for a specified time.</li>
</ul>
<p><strong>Architecture</strong></p>
<p>The Tzimulator framework is built on top of <a href="https://kubernetes.io/">Kubernetes</a> (also known as K8s), an open-source system for automating deployment of containerized applications. </p>
<p>Thanks to this highly automated framework, we can efficiently run different realistic simulations while tweaking various parameters. For testing block times, we have also run simulations on different hardware architectures. </p>
<p>All the experiments use machines with: </p>
<ul>
<li>6.5Gb of <span class="caps">RAM</span> memory,</li>
<li>100Gb persistent <span class="caps">SSD</span> storage,</li>
<li>Gigabit network with simulated network delays</li>
</ul>
<p>Meanwhile, the CPUs have been configured differently to reflect different hardware requirements. The experiment was carried out with 4 <span class="caps">CPU</span> cores of the following architectures:</p>
<ul>
<li><a href="https://gcloud-compute.com/n1-standard-4.html">n1-standard-4</a> (2Ghz),</li>
<li><a href="https://gcloud-compute.com/n2-standard-4.html">n2-standard-4</a> (2.6Ghz)</li>
</ul>
<p>In addition to the above parameters we can adjust the following:</p>
<ul>
<li><strong>Signing delay:</strong> Simulates signing times similar to slow hardware devices such as Ledger Nano S, remote signers, etc.</li>
<li><strong>Network delay:</strong> Simulates network congestions, with ad-hoc delay ranging from 20ms to 150ms.</li>
<li><strong>Mempool load:</strong> Defines target number of operations in the mempool at all times.</li>
<li><strong>Block time:</strong> The targeted time between blocks. Various block times were tested (15s, 10s, 8s, 5s) with variations of the above parameters and different versions of Octez to determine the <em>safety</em> of each configuration.</li>
</ul>
<p>We can also configure which data are collected for a given simulation run. For every node and baker we can recover standard Octez logs (<a href="https://tezos.gitlab.io/user/logging.html">daily-log</a>). For a comprehensive overview and analysis of node behavior, <a href="https://research-development.nomadic-labs.com/octez-metrics.html">Octez Metrics</a> can be enabled to feed a <a href="https://prometheus.io/">Prometheus</a> database with a <a href="https://gitlab.com/nomadic-labs/grafazos">Grafazos</a> dashboard. Finally, our consensus inspection tool <a href="https://research-development.nomadic-labs.com/introducing-teztale.html">Teztale</a> can be used for analyzing consensus performance.</p>
<p><strong>Differences with Mainnet</strong></p>
<p>Presently, the framework diverges from Mainnet in a few ways.</p>
<p>Node and baker configurations are uniform, with all instances using identical hardware, signing hardware, and docker images. </p>
<p>The framework also uses a reduced number of bakers (up to 250) and nodes (up to 250) compared to Mainnet’s larger count (around 400 bakers and 5000 nodes). All nodes operate within the same cluster, and network delay is instead simulated using a custom ad-hoc solution.</p>
<p>Regarding the network topology, the framework has a simpler setup compared to the Mainnet network. It uses the default <span class="caps">P2P</span> configuration, with each node connected to ~50 other nodes. The resulting graph for 250 nodes is not strongly connected, and some hops are needed for operations from the injection to reaching the farthest node.</p>
<table border="0">
<tr>
<td width="15%"></td>
<td><img alt="Image showing a simplified test network map" src="/images/blocktimereduction/testnetwork.png" width="100%"/></td>
<td width="15%"></td>
</tr>
</table>
<p><strong>The workflow: Identifying and fixing bottlenecks</strong></p>
<p>Pinpointing bottlenecks and implementing solutions was carried out as an iterative process:</p>
<ol>
<li>Choose block time and other parameters.</li>
<li>Conduct an experiment with the network under full load.</li>
<li>Analyze data collected from an internally developed profiler, <a href="https://research-development.nomadic-labs.com/octez-metrics.html">Octez Metrics</a>, and <a href="https://research-development.nomadic-labs.com/introducing-teztale.html">Teztale</a>.</li>
<li>Identify potential bottlenecks based on analysis of the collected data.</li>
<li>Develop an optimization and conduct additional experiments using the same parameters and the newly implemented improvement.</li>
<li>If the optimization demonstrates improved results, merge it into the Tezos codebase, and re-initiate the process.</li>
</ol>
<table border="0">
<tr>
<td width="15%"></td>
<td><img alt="Image showing the test approach as a circular process" src="/images/blocktimereduction/testprocess.png" width="100%"/></td>
<td width="15%"></td>
</tr>
</table>
<h2 id="the-baseline"><a class="toclink" href="#the-baseline">The baseline</a></h2>
<p>We started out with a baseline experiment: a network configured with the parameters below, notably a blocktime of 15 seconds which is equivalent to current Mainnet.</p>
<ul>
<li>Node version: <a href="https://gitlab.com/tezos/tezos/-/releases/v18.1">Octez v18.1</a></li>
<li>Block time: 15 seconds</li>
<li>Simulated signing delay: 1 second</li>
<li>Random network delay: 20-150ms</li>
<li>Number of nodes (each with an associated baker): 250</li>
<li>Target mempool operations: 2500</li>
<li>Hardware:<ul>
<li>6.5Gb <span class="caps">RAM</span></li>
<li>4 <span class="caps">CPU</span> cores</li>
<li>n1-standard-4 machine with Intel Broadwell <span class="caps">CPU</span> platform</li>
<li>PersistentSSD disk type</li>
</ul>
</li>
</ul>
<p>The summary report for the experiment shows that out of the 363 levels, only one level was found to be at round 1 where the rest were at round 0. </p>
<p>We observed that <em>quorum</em> – 66% attestation required for completing a block level – was reached in just over 11 seconds on average. Average time for block application (validation time + application time) was 6 seconds.</p>
<table cellpadding="10">
<tr>
<td style="border: 1px solid black;" colspan="2"><strong>Average time elapsed since block timestamp, high network load (full blocks)</strong>
</td>
</tr>
<tr>
<td style="border: 1px solid black;"><strong>Stages of a round</strong>
</td>
<td style="border: 1px solid black;"><strong>Baseline results</strong>
</td>
</tr>
<tr>
<td style="border: 1px solid black;">Block validated: ready for consensus
</td>
<td style="border: 1px solid black;">4.51s
</td >
</tr>
<tr>
<td style="border: 1px solid black;">Block fully applied: new chain head
</td>
<td style="border: 1px solid black;">6.02s
</td>
</tr>
<tr>
<td style="border: 1px solid black;">Pre-quorum reached: first consensus vote complete
</td>
<td style="border: 1px solid black;">7.47s
</td>
</tr>
<tr>
<td style="border: 1px solid black;">Quorum reached: second and final consensus vote complete
</td>
<td style="border: 1px solid black;">11.14s
</td>
</tr>
</table>
<p>The graph below (‘Attestation Reception Delay’ from <a href="https://research-development.nomadic-labs.com/introducing-teztale.html#the-level-page">Teztale’s Level page</a>) shows the result from a single, random block level during the experiment. It aligns with the summary, but also reveals that the (pre-)attestation receptions have a noticeable difference between the first and last occurrences. </p>
<p><img alt="Graph showing results from baseline experiment" src="/images/blocktimereduction/baselinegraph.png" width="100%"/></p>
<p>Overall, our observations confirmed that the network was safe with a 15 second block time. On average, quorum was reached after 11.14 seconds, with a maximum time of 12.46 seconds. </p>
<p>However it also confirmed that improvements would be required in order to lower block time to 10 seconds.</p>
<h2 id="implemented-improvements"><a class="toclink" href="#implemented-improvements">Implemented improvements</a></h2>
<p>Providing a stepwise walkthrough of every performed experiment and subsequent implementation of improvements would make this blog post_ very_ long. </p>
<p>Rather, we present an overview of all improvements implemented through iterations of the process described above. We also present the results of an experiment run with all improvements implemented. </p>
<p>The improvements consist of changes to the Octez baker and the Octez node’s mempool handling. A good example of an improvement with a significant impact is the <a href="https://gitlab.com/tezos/tezos/-/merge_requests/10712">pre-emptive forging</a> implemented into the Octez baker. </p>
<p>Previously, the baker proposing the block after the current one would wait until the end of the current level to begin the process. But that is not necessary. </p>
<p>Proposing a block can be split into three parts: forging, signing and injecting. Forging and signing are the most time consuming parts, but they can be started before the block level has started.</p>
<p>Most blocks have “idle” waiting time for bakers after quorum is reached, and pre-emptive forging lets the next baker begin forging the next block during this time, so it is ready for injection at the start of the new level. The result is earlier propagation through the network. </p>
<p>With 15 second block time, this change led to quorum being reached ~2 seconds faster – a significant improvement.</p>
<p>Additional baker improvements:</p>
<ul>
<li><a href="https://gitlab.com/tezos/tezos/-/merge_requests/10957">Operations selection</a>. Bakers are prevented from wasting time on validating a manager operation that would always fail due to insufficient remaining gas in the block.</li>
<li><a href="https://gitlab.com/tezos/tezos/-/merge_requests/10958">Operations request mechanisms</a>. Bakers no longer fetch complete mempools from peers, which can delay consensus operations when the mempool is large.</li>
<li><a href="https://gitlab.com/tezos/tezos/-/merge_requests/10789">Consensus disk writes</a>. The number of on-disk writes when registering consensus votes are reduced.</li>
<li><a href="https://gitlab.com/tezos/tezos/-/merge_requests/10572">Optional state disk writes</a>. Consensus state is no longer written to disk at each block.</li>
<li><a href="https://gitlab.com/tezos/tezos/-/merge_requests/10554">Attestation injection time</a>. Attestations are sent immediately after pre-quorum has been detected, without waiting for block application.</li>
</ul>
<p>Mempool improvements:</p>
<ul>
<li><a href="https://gitlab.com/tezos/tezos/-/merge_requests/11114">Advertisement computation</a>. A change to how the mempool data structure is filled, which saves a lot of computation.</li>
<li><a href="https://gitlab.com/tezos/tezos/-/merge_requests/10833">Advertisement priority</a>. Avoids re-advertising non-consensus operations that have been advertised for a previous block.</li>
<li><a href="https://gitlab.com/tezos/tezos/-/merge_requests/10802">Classification of operations order</a>. Optimizations of mempool search reduces time spent by putting more time consuming elements last.</li>
</ul>
<p>Lastly, the number of cycles with metadata kept on disk for full and rolling nodes has been <a href="https://gitlab.com/tezos/tezos/-/merge_requests/11325">reduced</a> from 6 to 2. The result is a lighter storage allowing faster access.</p>
<h2 id="result-of-improvements"><a class="toclink" href="#result-of-improvements">Result of improvements</a></h2>
<p>Below we present the results of an experiment using a development version of Octez (v20-dev) with the above improvements enabled. We used the same hardware specifications and parameters as in the baseline experiment.</p>
<p>The perhaps most important metric – the time required to reach quorum, allowing the network to progress to the next block – decreased <em>significantly</em>, from ~11 to ~4.5 seconds.</p>
<table cellpadding="10">
<tr>
<td style="border: 1px solid black;" colspan="3" ><strong>Average time elapsed since block timestamp, high network load (full blocks)</strong><br>
Before Octez improvements (v18.1) and after (v20-dev)
</td>
</tr>
<tr>
<td style="border: 1px solid black;"><strong>Stages of a round</strong>
</td>
<td style="border: 1px solid black;"><strong>v18.1</strong>
</td>
<td style="border: 1px solid black;"><strong>v20-dev</strong>
</td>
</tr>
<tr>
<td style="border: 1px solid black;">Block validated: ready for consensus
</td>
<td style="border: 1px solid black;">4.51s
</td>
<td style="border: 1px solid black;">1.08s (-76%)
</td>
</tr>
<tr>
<td style="border: 1px solid black;">Block fully applied: new chain head
</td>
<td style="border: 1px solid black;">6.02s
</td>
<td style="border: 1px solid black;">2.53s (-58%)
</td>
</tr>
<tr>
<td style="border: 1px solid black;">Pre-quorum reached: first consensus vote complete
</td>
<td style="border: 1px solid black;">7.47s
</td>
<td style="border: 1px solid black;">2.88s (-61%)
</td>
</tr>
<tr>
<td style="border: 1px solid black;">Quorum reached: second and final consensus vote complete
</td>
<td style="border: 1px solid black;">11.14s
</td>
<td style="border: 1px solid black;">4.55s (-59%)
</td>
</tr>
</table>
<p>The ‘Attestation Reception Delay’ graph shows the result for a single, random block level during the Octez v20-dev experiment. It aligns with the summary and shows that (pre-)attestations now arrive quicker and more consistently around the average time.</p>
<p><img alt="Graph showing results after implementing the improvements" src="/images/blocktimereduction/improvementgraph.png" width="100%"/></p>
<p>Based on these results, we are confident that it is safe to reduce Tezos’ block time to 10 seconds, once the improvements are implemented in Mainnet nodes and bakers.</p>
<p>Our experiments showed even an 8 second block time to be safe, but the more conservative 10 seconds was chosen for the upcoming “P” protocol proposal for extra safety margin.</p>
<h2 id="hardware-recommendations-again"><a class="toclink" href="#hardware-recommendations-again">Hardware recommendations (again)</a></h2>
<p>The following hardware specifications match the ones used for our experiments. They are provided here as a guideline, but bakers are advised to perform their own testing.</p>
<ul>
<li>3 cores, 2 needed by the node and 1 needed by the baker (arm64 or amd64/x86-64)</li>
<li><span class="caps">8GB</span> <span class="caps">RAM</span></li>
<li><span class="caps">100GB</span> <span class="caps">SSD</span> storage (or similar I/O performance)</li>
<li>A low-latency reliable internet connection </li>
</ul>
<p>While it is possible to run the baker setup on a cloud platform, it may not be cost effective over the long term. Instances with specs similar to following (or higher) would work:</p>
<ul>
<li><a href="https://gcloud-compute.com/n1-standard-4.html">n1-standard-4 (2 GHz)</a> machine with Intel Broadwell <span class="caps">CPU</span> platform</li>
<li>6.5Gb <span class="caps">RAM</span> allocated to baker and node processes</li>
<li><span class="caps">100GB</span> Persistent disk <span class="caps">SSD</span> disk type</li>
</ul>
<h2 id="impact-on-mainnet"><a class="toclink" href="#impact-on-mainnet">Impact on Mainnet</a></h2>
<p>As mentioned initially, lowering the block time improves the latency of Tezos Layer 1, resulting in a smoother experience and faster finality.</p>
<p>Some parameters are affected by the change:</p>
<p><strong>Gas per block.</strong> Since the gas limit per block is correlated with block time, a 10 second block time means the gas limit will be lowered from 2.4M to 1.73M gas units. As there are more blocks pr. minute, throughput is unchanged. The hard gas limit per operation is left unchanged at 1M gas units.</p>
<p><strong>Consensus traffic.</strong> Increasing the number of blocks per minute increases bandwidth consumption from consensus operations. Experiments showed an increase from 200kB/s to 250kB/s on a network running at full load, which is well within acceptable limits.</p>
<p><strong>Cycle length.</strong> Increasing the number of blocks per minute (from 4 to 6) also impacts the cycle size as each cycle will have more blocks (from 16384 to 24576). </p>
<p>For nodes running a <a href="https://tezos.gitlab.io/user/history_modes.html">rolling history mode</a>, we’ve implemented a change that counteracts the increase in storage due to the extra blocks. Historically, such nodes have been required to store two weeks of blockchain data (5-6 cycles) for security purposes, but the deterministic finality of the Tenderbake consensus algorithm allows us to safely reduce it to 3-6 days (1-2 cycles). </p>
<p>Hence, while the number of blocks is increased by 50%, the storage footprint for rolling nodes is reduced by ~75%. Meanwhile, archive nodes and full nodes, which store all the chain’s history, will see an increased disk footprint with a 10 second block time.</p>
<h2 id="next-steps"><a class="toclink" href="#next-steps">Next steps</a></h2>
<p>We will continue to run experiments, identify bottlenecks and implement improvements to further improve overall performance of Octez and enable even lower block times.</p>
<p>A factor worth pointing out is the signing delay introduced by older hardware devices, such as the Ledger Nano S. Experiments with added delays corresponding to these hardware devices yielded the following results:</p>
<table cellpadding="10">
<tr>
<td style="border: 1px solid black;"><strong>Machine type</strong>
</td>
<td style="border: 1px solid black;"><strong>Signing device</strong>
</td>
<td style="border: 1px solid black;"><strong>Minimal safe block time</strong>
</td>
</tr>
<tr>
<td style="border: 1px solid black;" rowspan="2" ><a href="https://gcloud-compute.com/n1-standard-4.html">n1-standard-4</a><br>
(4-core vCPU @ 2GHz, <span class="caps">8GB</span> <span class="caps">RAM</span> assigned)
</td>
<td style="border: 1px solid black;">Ledger Nano S
</td>
<td style="border: 1px solid black;">8
</td>
</tr>
<tr>
<td>Ledger Nano S+
</td>
<td style="border: 1px solid black;">7
</td>
</tr>
<tr>
<td style="border: 1px solid black;" rowspan="2"><a href="https://gcloud-compute.com/n2-standard-4.html">n2-standard-4</a><br>
(4-core vCPU @ 2.6GHz, <span class="caps">8GB</span> <span class="caps">RAM</span> assigned)
</td>
<td style="border: 1px solid black;">Ledger Nano S
</td>
<td style="border: 1px solid black;">6
</td>
</tr>
<tr>
<td style="border: 1px solid black;">Ledger Nano S+
</td>
<td style="border: 1px solid black;">5
</td>
</tr>
</table>
<p>The results indicate that a further reduction of block time (by 1 second), could be achieved confidently, if the majority of bakers currently using Ledger Nano S were to adopt the Ledger Nano S+, or another equivalently faster signing solution.</p>
<p>Also worth noting is the difference in performance between the two machine types. Unsurprisingly, the ‘n2’ machine’s faster <span class="caps">CPU</span> enables significantly lower block times. However, hardware specifications must always be considered in the context of making sure Tezos has a low barrier for participation to encourage decentralization.</p>
<h2 id="keeping-tezos-decentralized"><a class="toclink" href="#keeping-tezos-decentralized">Keeping Tezos decentralized</a></h2>
<p>As our experiments show, Tezos is able to have a Layer 1 block time of 10 seconds while maintaining low hardware requirements.</p>
<p>A quick way to enable further block time reductions would be to increase requirements for hardware. This is not an uncommon approach for blockchains looking to boost performance.</p>
<p>Ethereum, for example, has higher <a href="https://ethereum.org/run-a-node#plan-on-staking">official hardware requirements</a> than Tezos – at least <span class="caps">16GB</span> <span class="caps">RAM</span> and a more performant <span class="caps">CPU</span>. A chain like Solana is able to produce blocks at a very high pace, but relies on significantly higher <a href="https://docs.solanalabs.com/operations/requirements#hardware-recommendations">official hardware requirements</a>, including 12 <span class="caps">CPU</span> cores @2.8 GHz, <span class="caps">256GB</span> of <span class="caps">RAM</span>, 2 <span class="caps">SSD</span> disks up to <span class="caps">1TB</span>, and 1 Gbit/s internet connection with 10Gbit/s preferred.</p>
<p>For now, we maintain a more conservative target for our work on Tezos. This is because higher barriers for participation may affect decentralization negatively – the economic cost of the hardware is one factor, with access to the required internet bandwidth being another. </p>
<p>As a reminder, decentralization is not just a principled ideal. Ultimately, decentralization remains the only way to enable true censorship resistance – a core purpose of blockchains.</p>PoS refinements and Private Rollups: Oxford 2 upgrade is live!2024-02-09T14:40:00+01:002024-02-09T14:40:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2024-02-09:/oxford-is-live.html<p>Tezos has activated its 15th protocol upgrade.</p><p>On February 9 2024 13:23:45 <span class="caps">UTC</span>, the Tezos blockchain successfully activated the Oxford 2 protocol upgrade at block <a href="https://tzkt.io/5070849">#5,070,849</a>.</p>
<p>This 15th upgrade was jointly developed by Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda <span class="amp">&</span> Functori.</p>
<p>Oxford 2 contains the following changes:</p>
<ul>
<li><a href="https://research-development.nomadic-labs.com/oxford2-announcement.html#automated-staking">Refinement of Tezos PoS:</a> the Oxford 2 proposal introduces changes to <a href="https://research-development.nomadic-labs.com/oxford2-announcement.html#refined-slashing">slashing</a> and <a href="https://research-development.nomadic-labs.com/oxford2-announcement.html#automated-staking">an automated staking mechanism</a> for bakers. The latter aims to smoothen the transition towards a new staked funds management <span class="caps">API</span> and avoids manual bookkeeping to counter over-delegation.</li>
<li><a href="https://research-development.nomadic-labs.com/oxford2-announcement.html">Private rollups</a>: Oxford 2 introduces private Smart Rollups, allowing developers to choose between permissionless or permissioned deployments. Additionally, Oxford 2 <a href="https://research-development.nomadic-labs.com/oxford2-announcement.html#introducing-private-rollups-and-other-improvements-to-smart-rollups">simplifies the deployment of rollups</a> both on protocol and periodic test networks, as well as on ad-hoc dedicated ones.</li>
<li><a href="https://research-development.nomadic-labs.com/oxford2-announcement.html#timelocks-are-re-enabled">Timelocks are re-enabled:</a> a new design and implementation of Timelocks addresses security concerns that led to their temporary deactivation in a previous protocol upgrade.</li>
</ul>
<p>For more details, see the <a href="https://research-development.nomadic-labs.com/oxford2-announcement.html">Oxford 2 announcement post</a>. </p>
<p>A deeper technical description can be found in the protocol proposal’s <a href="https://tezos.gitlab.io/oxford/protocol.html">technical documentation</a>, and a complete list of changes is provided in <a href="https://tezos.gitlab.io/protocols/018_oxford.html">Oxford 2’s changelog</a>.</p>
<h2 id="big-things-coming"><a class="toclink" href="#big-things-coming">Big things coming</a></h2>
<p>The activation of Oxford 2 is just the beginning of a big 2024 for Tezos. We are excited to be part of the ongoing work to evolve Tezos – and the blockchain space.</p>
<p><strong>Data-Availability Layer (<span class="caps">DAL</span>) on testnet</strong>. The <span class="caps">DAL</span> is currently live on Tezos’ Weeklynet, and <a href="https://research-development.nomadic-labs.com/dal-testers.html">we encourage bakers and rollup operators</a> to help us test this cornerstone of Tezos’ future. To understand what it does, and what it means for Tezos, see our <a href="https://research-development.nomadic-labs.com/data-availability-layer-tezos.html">introduction to the <span class="caps">DAL</span></a>.</p>
<p><strong>Get ready for the launch of Etherlink</strong>. <a href="https://www.etherlink.com/">Etherlink</a> will be an <span class="caps">EVM</span>-compatible optimistic rollup with high throughput, a decentralized sequencer, low gas fees, and <span class="caps">MEV</span> protection, powered by Tezos <a href="https://tezos.com/developers/smart-rollups/">Smart Rollups</a>. Check out <a href="https://www.etherlink.com/">etherlink.com</a> to learn more and get involved.</p>
<p>Meanwhile, we are busy working on an upcoming protocol upgrade proposal, “P”. Stay tuned!</p>Introducing Private Smart Rollups on Tezos2024-01-30T16:00:00+01:002024-01-30T16:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2024-01-30:/private-rollups.html<p>Smart Rollup builders can now choose their desired level of decentralization.</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: A new whitelisting feature allows Smart Rollup builders to choose their desired level of decentralization.</strong></p>
<p>The majority of blockchains don’t provide transaction privacy by default. Tezos is no exception. An address is a pseudonym, and transactions are public. A resourceful and determined actor will often be able to connect an address to a real identity, and then all transaction history is freely available.</p>
<p>This is the case whether you are transacting on Layer 1, in an optimistic rollup, or in a zk-rollup (also called validity rollup). By default there is no privacy and special solutions are required to achieve it.</p>
<p>So how can we achieve privacy in <a href="https://tezos.com/developers/smart-rollups/">Smart Rollups</a> for use cases that require it?</p>
<p>A common way to achieve privacy is to use <em>zero-knowledge proofs</em>, also called zk-proofs<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>. In short they enable you to prove properties about some data without revealing the data itself. Zk-proofs have been adopted by many blockchain projects, <a href="https://opentezos.com/tezos-basics/sapling/">including Tezos’ Layer 1</a>, in order to provide optional privacy for simple transactions. However, using zk-proofs quickly becomes complicated and resource intensive when used for complex transactions and smart contracts with many participants. </p>
<p>Some techniques exist that can be applied, such as <em>multi-party computation</em> where a group of participants perform collaborative computation over pooled data while preserving confidentiality for individual inputs. But this is a complex solution that relies on advanced cryptography. It’s costly to implement, and it doesn’t scale well. </p>
<p>So, is there not a simpler way to achieve privacy in an optimistic rollup? There is, but it requires us to relax some security assumptions, and it is only useful for some specific use cases.</p>
<h2 id="trading-some-decentralization-for-privacy"><a class="toclink" href="#trading-some-decentralization-for-privacy">Trading (some) decentralization for privacy</a></h2>
<p>The Nomadic Labs adoption team has worked with financial institutions with specific requirements for privacy, and we have developed a simple solution that satisfies their needs. It is a Smart Rollup, but we relax assumptions regarding 1) publication of transaction data, and 2) decentralization of the rollup.</p>
<p>Transaction data for the rollup is not posted on Layer 1 or otherwise publicly, and Layer 1 provides no guarantees about its availability for later verification. Instead transaction data is posted to a non-public <a href="https://research-development.nomadic-labs.com/introducing-data-availability-committees.html">Data-Availability Committee</a> (<span class="caps">DAC</span>). A <span class="caps">DAC</span> is typically run by a consortium of interested parties, which retain control over the data. This enables privacy for incoming transaction data at the cost of verifiability for outsiders.</p>
<p>Imagine a consortium of banks that use a Smart Rollup for fast and easily auditable settlement. As the transaction data is confidential, it is posted to a non-public <span class="caps">DAC</span>. The only thing made public are <em>commitments </em>– hash values representing the rollup’s state that reveal nothing about the underlying data.</p>
<p>Using a <span class="caps">DAC</span> has been possible on Tezos for a while, but it isn’t enough for keeping a rollup private. The permissionless nature of Smart Rollups means that anyone could create challenges to rollup commitments, even honest ones, which could force rollup operators to reveal some data publicly or face an economic penalty. Since only parties with access to the <span class="caps">DAC</span> can meaningfully challenge commitments in the first place, it does not make sense to let anyone do so.</p>
<p>For this reason, the <a href="https://research-development.nomadic-labs.com/oxford2-announcement.html">Oxford 2 protocol proposal</a> contains a small change to the protocol allowing Smart Rollup developers to define a whitelist of allowed operators (and hence potential challengers). It’s a simple and private solution for a consortium-style rollup that comes with the benefits of being secured by Tezos consensus, and having a direct bridge to Layer 1 and the wider Tezos ecosystem.</p>
<p>To be clear, implementing a whitelist effectively makes the Smart Rollup <em>permissioned</em> – not fully decentralized. Having it as an option lets Smart Rollup builders choose their desired level of decentralization based on their particular use case. </p>
<h2 id="how-smart-rollup-whitelists-work"><a class="toclink" href="#how-smart-rollup-whitelists-work">How Smart Rollup whitelists work</a></h2>
<p>Whether a Smart rollup is private (permissioned) or public (permissionless) is defined at deployment. If a <a href="https://tezos.gitlab.io/alpha/smart_rollups.html#private-rollups">whitelist</a> is defined, the rollup is considered private by the protocol. If not, it is considered public. </p>
<p>For a private rollup, only addresses on the whitelist can be operators for the rollup, including publishing commitments, challenging commitments, and participating in <a href="https://tezos.gitlab.io/alpha/smart_rollups.html">refutation games</a>.</p>
<p>It is the responsibility of the rollup <a href="https://tezos.gitlab.io/shell/smart_rollup_node.html#developing-wasm-kernels">kernel</a> to maintain the whitelist. It can be updated with a specific outbox message, which includes a new whitelist replacing the existing one. Kernels must therefore implement their own access control logic to add and remove addresses.</p>
<p>Note that a private rollup can become public, but not the other way around.</p>
<ul>
<li><strong>Private rollups becoming public</strong>. If the specified outbox message is sent without containing a whitelist, the rollup becomes public. If a rollup is never meant to become public, its kernel can be designed to disallow empty whitelist messages.</li>
<li><strong>Public rollups cannot become private</strong>. An outbox message with a whitelist for a public rollup will fail. This is to prevent a rollup becoming private after it has been advertised as public.</li>
</ul>
<h2 id="more-control-for-builders-and-testers"><a class="toclink" href="#more-control-for-builders-and-testers">More control for builders and testers</a></h2>
<p>The whitelisting feature doesn’t just benefit those working with confidential data.</p>
<p>Some developers may prefer to initially deploy a “beta” rollup with a whitelist, even if still on testnet, and only later make it public. This kind of “training wheels” approach is often used for launching blockchain projects.</p>
<p>Others may opt for a permanent semi-private setup, which can reduce risk for the rollup operator. The transaction data are available for all to check, but only whitelisted operators can initiate refutation games. Transparency is kept, but bad-faith actors can’t leverage kernel bugs or errors by the rollup operators to cause problems, such as incurring a loss of the 10,000 tez stake required to operate a rollup.</p>
<p>With the whitelisting feature, builders have the option to deploy rollups with the level of decentralization that fits their use case, whether it has to do with privacy, security or simply comfort.</p>
<p>We welcome any kind of feedback that helps us make Tezos the best possible solution for blockchain builders. If you are interested in launching a project on Tezos and have questions or suggestions, don’t hesitate to <a href="https://www.nomadic-labs.com/contact-us/">reach out</a>.</p>
<!-- Footnotes themselves at the bottom. -->
<h2 id="notes"><a class="toclink" href="#notes">Notes</a></h2>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>Zk-rollups use elements from zero-knowledge technology, but the “zk” name is slightly misleading as the technology is implemented in a way that provides no more privacy by default than Layer 1 or optimistic rollups. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>The Data-Availability Layer is Coming to Tezos – Now on Testnet2024-01-25T16:00:00+01:002024-01-25T16:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2024-01-25:/dal-testers.html<p>Test the <span class="caps">DAL</span> on Tezos’ Weeklynet.</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: The Data-Availability Layer is live on Tezos’ Weeklynet, and
we call on bakers and rollup operators to help us test this
groundbreaking “rollup booster”. This blog post shows you how to get started.</strong></p>
<p>In a previous blog post, we introduced the Data-Availability Layer
(<span class="caps">DAL</span>) as a game-changing “rollup booster” for Tezos. If you’re new to
the <span class="caps">DAL</span> concept, we highly recommend starting with <a href="https://research-development.nomadic-labs.com/data-availability-layer-tezos.html">this blog
post</a>. For
a more technical walk-through, see <a href="https://www.youtube.com/watch?v=Mjn0BH7ie4E">this
presentation</a> from the
TezDev 2023 conference.</p>
<p>In a nutshell, the <span class="caps">DAL</span> is a permissionless peer-to-peer (<span class="caps">P2P</span>) network
which is part of the Tezos protocol and runs in parallel with
Layer 1. It allows for rollup transaction data (Layer 2) to be
published outside of the confines of Layer 1 blocks. Layer 1 bakers
continuously monitor the <span class="caps">DAL</span> and attest on Layer 1 whether a given
piece of data is available on the <span class="caps">DAL</span>.</p>
<h2 id="whats-happening"><a class="toclink" href="#whats-happening">What’s happening?</a></h2>
<p>We expect the <span class="caps">DAL</span> to go live on Mainnet in 2024, which is why we are
now publishing this call for testers on Weeklynet.</p>
<p>Once the <span class="caps">DAL</span> goes live on Mainnet, bakers will have to set up a <span class="caps">DAL</span>
node in addition to the baker daemon. It’s important to test this step
ahead of Mainnet activation, as <span class="caps">DAL</span> participants will be launching a
new <span class="caps">P2P</span> network. To be clear: testing distributed networks is hard,
and nothing can compete with a live, realistic testnet.</p>
<p>Additionally, we need feedback from rollup operators. The <span class="caps">DAL</span> offers
rollup operators a permissionless, out-of-the-box data availability
solution, enabling them to publish large amounts of data for a rollup
at a very low cost and without sacrificing decentralization. Of
course, success for such a system is contingent upon a great developer
experience. We will need fresh eyes on this process and feedback to
help us make it the best possible experience.</p>
<h2 id="getting-started"><a class="toclink" href="#getting-started">Getting started</a></h2>
<p>To get you up and running, we have recently published several resources:</p>
<ul>
<li><a href="https://tezos.gitlab.io/shell/dal.html">Technical documentation</a>
for the <span class="caps">DAL</span></li>
<li>A tutorial explaining <a href="https://docs.tezos.com/tutorials/join-dal-baker">how to easily setup a <span class="caps">DAL</span> node</a></li>
<li>A tutorial demonstrating <a href="https://docs.tezos.com/tutorials/build-files-archive-with-dal">how a Smart Rollup can use the
<span class="caps">DAL</span></a></li>
</ul>
<p>These documents should be enough to get you started. Afterward, the
sky is the limit: you decide which data will be published on the <span class="caps">DAL</span>
and for what purposes.</p>
<p>The <span class="caps">DAL</span> is a cornerstone for the future of Tezos, including
innovations such as <a href="https://www.etherlink.com/">Etherlink</a> and <a href="https://www.youtube.com/watch?v=6USr8iqXXFE">Tezos
2.0</a>. That is why your
feedback is very important. We encourage anyone involved with testing
these features to share their feedback in a comment on <a href="https://forum.tezosagora.org/t/call-for-testers-the-dal-is-live-on-weeklynet/6000">this Tezos
Agora post</a>.</p>
<h2 id="next-steps"><a class="toclink" href="#next-steps">Next steps</a></h2>
<p>We are planning to introduce the <span class="caps">DAL</span> with a phased approach. More
precisely, the rollout plan for the <span class="caps">DAL</span> — assuming the community
embraces it — is to:</p>
<ol>
<li>First enable the <span class="caps">DAL</span> on Mainnet without baker incentives and with
optional participation.</li>
<li>Later introduce baker incentives for the <span class="caps">DAL</span>, at which point <span class="caps">DAL</span>
participation becomes an integrated part of the baker role.</li>
</ol>
<p>The rationale is that we want to give bakers ample time to familiarize
themselves with the <span class="caps">DAL</span> and validate its functionality before
integrating incentives on Mainnet, as the incentives touch on a key
part of Tezos Layer 1: the baker role and related economics/rewards.</p>
<p>We intend to publish a design proposal specifically for baker
incentives around the time when we propose the protocol enabling <span class="caps">DAL</span>,
along with an initial implementation live on Weeklynet. This should
give plenty of time for bakers to provide feedback on these aspects too.</p>
<p>The <span class="caps">DAL</span> is a key element in keeping Tezos a technological
frontrunner. Proper testing of it is vital, and we look forward to
working with the broader Tezos community to make it ready for Mainnet.</p>Announcing Oxford 2: a revised 15th protocol upgrade proposal2023-12-05T10:00:00+01:002023-12-05T10:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2023-12-05:/oxford2-announcement.html<p>Refined Tezos PoS, new Timelocks and more!</p><p><em>This is a joint post from Nomadic Labs, Marigold, TriliTech, Oxhead
Alpha, Tarides, DaiLambda <span class="amp">&</span> Functori.</em></p>
<p>After the <a href="https://research-development.nomadic-labs.com/oxford-announcement.html">Oxford protocol upgrade
proposal</a>
did not gather the required supermajority of affirmative votes in its
promotion period vote, we <a href="https://forum.tezosagora.org/t/towards-a-revised-oxford-proposal/5808">presented an action
plan</a>
centered around improving staking <span class="caps">UX</span> for bakers and furthering the
continuous development of Smart Rollups and other features of the
Tezos protocol, e.g. Timelocks.</p>
<p>We thank bakers, infrastructure and application developers, and the
Tezos ecosystem at large for their feedback, in particular <a href="https://forum.tezosagora.org/t/feedback-request-slashing-and-automated-staking-for-bakers-in-the-oxford-2-proposal/5836">towards
the proposed
changes</a>
in slashing and automated staked funds management for bakers.</p>
<p>Today, we are pleased to announce that this revised Oxford 2 protocol
proposal is ready to be submitted to the Tezos governance
mechanism. The proposal’s hash is
<code>ProxfordYmVfjWnRcgjWH36fW6PArwqykTFzotUxRs6gmTcZDuH</code>.</p>
<p>In a nutshell, the proposed changes in <strong>Oxford 2</strong> can be summarized
as follows:</p>
<ul>
<li><a href="#from-oxford-to-oxford-2-adaptive-issuance-and-staking"><strong>Adaptive Issuance and the new Staking mechanism are
disabled:</strong></a>
Oxford 2 would not allow the activation of <a href="https://research-development.nomadic-labs.com/adaptive-issuance.html#adaptive-issuance">Adaptive
Issuance</a>
nor extend the
<a href="https://research-development.nomadic-labs.com/adaptive-issuance.html#new-staking-mechanism">Staking</a>
mechanism to <em>delegators</em> in its lifetime. The feature vote
mechanism inherited from Oxford remains available, but the result of
the vote is ignored by the Oxford 2 protocol.</li>
<li><a href="#refined-slashing"><strong>Refinement of Tezos PoS:</strong></a> the Oxford 2
proposal introduces changes to <a href="#refined-slashing">slashing</a> and <a href="#automated-staking">an
automated auto-staking mechanism</a> for
bakers. The latter aims to smooth the transition towards the new
staked funds management <span class="caps">API</span> and avoids manual bookkeeping to counter over-delegation.</li>
<li><a href="#timelocks-are-re-enabled"><strong>Timelocks are re-enabled:</strong></a> a new
design and implementation of Timelocks addresses security concerns
that led to their temporary deactivation in a previous protocol upgrade.</li>
<li><a href="#introducing-private-rollups-and-other-improvements-to-smart-rollups"><strong>Smart Rollups keep
evolving:</strong></a>
Oxford 2 introduces <em>private</em> Smart Rollups, allowing developers to
choose between permissionned or permissionless deployments. Oxford 2
also simplifies the deployment of rollups both on protocol and
periodic test networks, as well as on ad-hoc dedicated ones.</li>
</ul>
<p>In the sequel, we provide further detail on the main features of the
Oxford 2 protocol upgrade proposal, and highlight key changes with
respect to its predecessor, Oxford.</p>
<p>A complete summary of changes is available in <a href="https://tezos.gitlab.io/protocols/018_oxford.html">Oxford 2’s
Changelog</a>.</p>
<h2 id="from-oxford-to-oxford-2-adaptive-issuance-and-staking"><a class="toclink" href="#from-oxford-to-oxford-2-adaptive-issuance-and-staking">From Oxford to Oxford 2: Adaptive Issuance and Staking</a></h2>
<p>First and foremost, the Oxford 2 proposal <strong>does not</strong> enable the
activation of Adaptive Issuance, nor allows delegators to become
<em>stakers</em>. A <strong>disabled</strong> feature flag has been introduced, which
would prevent these features from activating in the protocol’s lifetime.</p>
<p>The <strong>disabled</strong> feature flags signal our position that the feature is
not yet mature enough to be activated on Tezos Mainnet, and we will
continue working together with infrastructure builders to ensure that
both the feature and the required ecosystem tooling support are ready soon.</p>
<p>However, the <a href="https://research-development.nomadic-labs.com/oxford-announcement.html#activating-adaptive-issuance-and-staking">feature activation
vote</a>
introduced in the original Oxford proposal is still available to
bakers: participation is still recorded by the Tezos protocol, but it
does not take effect and can never activate the feature. We will reset
the <a href="https://tezos.gitlab.io/oxford/adaptive_issuance.html#feature-activation-vs-protocol-activation">associated
<span class="caps">EMA</span></a>
to 0 in any future protocol proposal developed by us that enables the
feature flag.</p>
<p>The rationale behind this choice is twofold: first, bakers can
manifest their attitude towards Adaptive Issuance and the new Staking
mechanism in an informal manner, which is a useful signal. Second,
this is a conservative approach which causes less friction to ongoing
efforts to develop and test support for these features.</p>
<h3 id="automated-staking"><a class="toclink" href="#automated-staking">Automated Staking</a></h3>
<p>The Oxford 2 proposal introduces an <a href="https://docs.google.com/document/d/1ND9CMiIrPYJmM2svJwq4wUbDwdszAZ2Pb5Bo4P220xw/edit#heading=h.wkalagp39pmw">automated staking
mechanism</a>
for bakers aiming to smooth the transition between the current <em>frozen
deposits</em> management system in Tezos Mainnet, and the <a href="https://research-development.nomadic-labs.com/adaptive-issuance.html#staked-funds-management">staked funds
management
mechanism</a>
originally proposed in Oxford.</p>
<p>The rationale is that the resulting mechanism should not entail
noticeable changes for baker operations. In a nutshell,</p>
<ul>
<li>The <code>set_deposit_limit</code> operation is preserved – to enable bakers to
continue capping their deposit limit further than the default value
of 10% of their staking balance</li>
<li>The <em>manual</em> <a href="https://research-development.nomadic-labs.com/adaptive-issuance.html#staked-funds-management">staked funds management
<span class="caps">API</span></a>
implemented in Oxford is disabled, and the protocol automatically
adjusts frozen deposits to match the chosen deposit limit at the end
of each cycle.</li>
</ul>
<p>See <a href="https://docs.google.com/document/u/0/d/1uYjcWR7tOBB7EaPb3JHDsedqiZ4SOKOiuDMVMqX6Bg0/edit">this design
document</a>
for further technical details of the proposed mechanism – and don’t
hesitate to reach out to us on <a href="https://forum.tezosagora.org/t/feedback-request-slashing-and-automated-staking-for-bakers-in-the-oxford-2-proposal">this Tezos Agora
thread</a>
with questions or feedback.</p>
<h3 id="refined-slashing"><a class="toclink" href="#refined-slashing">Refined slashing</a></h3>
<p>In Oxford 2, there are also a few changes to <a href="https://docs.google.com/document/d/1GLe8B0s7A88Av2rE5Ezumf8rJ7I7f-hNzJysQsaLjgE/edit#heading=h.2ulqx5s1127r">slashing penalties for
double
signing</a>
– that is, for behaviors that might threaten the safety and liveness
of the Tezos ledger. As originally proposed in Oxford, the penalties
for <strong>double baking</strong> are made proportional to the baker’s staking
balance. However, Oxford 2 sets the penalty at <strong>5%</strong> of the stake,
instead of 10% as set in Oxford – or the 640 tez as it is currently
the case on Tezos Mainnet.</p>
<p>Oxford 2 also implements changes to how and when slashing penalties
are applied by the Tezos economic protocol:</p>
<ul>
<li>Slashing penalties take effect at the end of a blockchain cycle
instead of immediately after the inclusion of the corresponding
denunciation operation. They are applied <strong>after</strong> the distribution
of the cycle’s attestation rewards and <strong>before</strong> the computation of
new voting and baking rights.</li>
<li>Balance updates from slashing penalties are published in the block
ending the cycle, including burned tez and denunciation rewards.</li>
<li>Repeated offending bakers (those who got 51% of their stakes slashed
over 2 cycles), will be marked as <strong>forbidden</strong> at the inclusion of
the triggering denunciation, immediately preventing them from
participating in consensus for the rest of the cycle — when their
status would be reassessed using the same criteria.</li>
</ul>
<p>See <a href="https://docs.google.com/document/d/1GLe8B0s7A88Av2rE5Ezumf8rJ7I7f-hNzJysQsaLjgE/edit">Slashing and you: a
primer</a>
for further detail, including the effect of slashing on frozen
deposits. Again, don’t hesitate to give feedback or ask questions <a href="https://forum.tezosagora.org/t/feedback-request-slashing-and-automated-staking-for-bakers-in-the-oxford-2-proposal">on
Tezos
Agora</a>.</p>
<h3 id="further-readings"><a class="toclink" href="#further-readings">Further reading(s)</a></h3>
<p>Oxford 2 introduces further minor changes that might impact bakers and
indexers. They are summarized in <a href="https://docs.google.com/document/d/1l6t51j6gSn6oO1C6B121nM44UoowNIrkG4NqcexI0Xg">this
document</a>.</p>
<h2 id="timelocks-are-re-enabled"><a class="toclink" href="#timelocks-are-re-enabled">Timelocks are re-enabled</a></h2>
<p>As originally planned for Oxford, the Michelson instructions providing
support for cryptographic
<a href="https://tezos.gitlab.io/oxford/timelock.html">Timelocks</a> are
re-enabled in Oxford 2.</p>
<p>Timelocks were disabled in the <a href="https://research-development.nomadic-labs.com/lima-is-live.html">Lima protocol
upgrade</a>,
following <a href="https://forum.tezosagora.org/t/vulnerability-found-in-the-timelock-feature/4660">the discovery of a security
flaw</a>
at the application level. This error could plausibly allow an
adversary to play two roles at the same time in the Timelock protocol
to bypass the security of the system, acquiring the secret. As a
result, the attacker would be able to create and post false proofs and
unlock the Timelock chest.</p>
<p>The feature has since been redesigned<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>, and after thorough testing,
including an external audit performed by <a href="https://inference.ag/">Inference
<span class="caps">AG</span></a>, we are ready to re-activate this feature
in the Tezos protocol.</p>
<p>Additionally, the new version of Timelock features <a href="https://tezos.gitlab.io/oxford/cli-commands.htmlhttps://tezos.gitlab.io/oxford/cli-commands.html">new client <span class="caps">CLI</span>
commands</a>
allowing users to create, open, and verify time-locked chests, as well
as to execute the pre-computations required for fast chest generation.</p>
<h2 id="introducing-private-rollups-and-other-improvements-to-smart-rollups"><a class="toclink" href="#introducing-private-rollups-and-other-improvements-to-smart-rollups">Introducing <em>private</em> rollups — and other improvements to Smart Rollups</a></h2>
<p>Oxford 2 introduces a new complementary feature for <a href="https://tezos.com/developers/smart-rollups/">Smart
Rollups</a>: the ability to
deploy <em>private</em> rollups. They allow for defining a <em>whitelist</em> of
allowed rollup operators, which is maintained by the rollup kernel and
enforced by the Tezos protocol.</p>
<p>Note that this new feature does not affect the existing permissionless
approach — which will continue to be the default rollup behavior. It
is rather a complementary choice for operators, broadening the
spectrum of potential adopters of Tezos Layer 2 solutions.</p>
<p>Indeed, this feature was developed in response to requests by
potential adopters of Tezos Smart Rollups, whose data privacy needs
are incompatible with permissionless fraud proofs, typically where
user data cannot be shared openly. By restricting the participation in
refutation games to whitelisted operators, private rollups prevent
potentially sensitive data from being leaked in the generation of
fraud proofs.</p>
<p>We will provide further insights into this new feature in an upcoming
blog post.</p>
<p>Oxford 2 also includes several <em>quality-of-life</em> improvements for
Smart Rollup developers, <a href="https://research-development.nomadic-labs.com/oxford-announcement.html#improvements-to-smart-rollups">inherited from the Oxford
proposal</a>:</p>
<ul>
<li>The rollup origination operation has been simplified, and no longer
requires a so-called_ “origination proof”_ as an argument. This
should benefit projects building on top of the Smart Rollups
infrastructure stack.</li>
<li>Similarly to what was already possible for smart contracts, it is
now possible to hard-code the origination of Smart Rollups in a
chain’s genesis block. This is used, for example, in the deployment
of <a href="https://research-development.nomadic-labs.com/evm-tezos-testnet.html"><span class="caps">EVM</span> rollups on test
networks</a>.</li>
<li>A new <span class="caps">WASM</span> <span class="caps">PVM</span> revision is released (<code>2.0.0-r3</code>). It enriches the
set of host functions available for querying the durable storage of
a Smart Rollup, and introduces new capabilities aimed at integrating
with the upcoming Tezos’
<a href="https://research-development.nomadic-labs.com/data-availability-layer-tezos.html"><span class="caps">DAL</span></a>. Existing
rollups will see their <span class="caps">PVM</span> automatically upgrade, and newly
originated Smart Rollups will use this version directly.</li>
</ul>
<h2 id="looking-ahead-to-2024"><a class="toclink" href="#looking-ahead-to-2024">Looking ahead to 2024</a></h2>
<p>Oxford 2 comes at the end of an exciting year for Tezos:</p>
<ul>
<li>Smart Rollups went live on Mainnet <a href="https://research-development.nomadic-labs.com/Mumbai-is-live.htmlhttps://research-development.nomadic-labs.com/Mumbai-is-live.html">when Mumbai
activated</a>,
a crucial milestone in our roadmap for scalability.</li>
<li>We
<a href="https://research-development.nomadic-labs.com/million-tps-demo.html">demonstrated</a>
that Tezos can support – and surpass – the 1M <span class="caps">TPS</span> throughput benchmark.</li>
<li>Mumbai also <a href="https://research-development.nomadic-labs.com/mumbai-preview.html#pipelining-is-fully-deployed-giving-us-15-second-block-times">halved blocktime from 30s to
15s</a>,
after the Pipelining project completed its full-revamp of the
internal block and (operation) validation logic.</li>
<li><a href="https://research-development.nomadic-labs.com/Nairobi-is-live.html">Nairobi</a>
increased overall performance on Tezos Layer 1, reducing <a href="https://research-development.nomadic-labs.com/nairobi-announcement.html#increased-tps-thanks-to-a-finer-grained-gas-modelhttps://research-development.nomadic-labs.com/nairobi-announcement.html#increased-tps-thanks-to-a-finer-grained-gas-model">gas costs
for common
operations</a>
and enabling faster <a href="https://research-development.nomadic-labs.com/nairobi-announcement.html#fine-tuned-validation-pipelining-for-faster-consensus">propagation of consensus
operations</a>.</li>
<li>Data-availability solutions continued their development, with <a href="https://research-development.nomadic-labs.com/introducing-data-availability-committees.htmlhttps://research-development.nomadic-labs.com/introducing-data-availability-committees.html">DACs
being
finalized</a>,
and <a href="https://research-development.nomadic-labs.com/data-availability-layer-tezos.html">the
<span class="caps">DAL</span></a>
getting closer and closer to Tezos Mainnet.</li>
<li><a href="https://www.etherlink.com/">Etherlink</a>, an <span class="caps">EVM</span>-compatible rollup,
is available <a href="https://forum.tezosagora.org/t/a-new-etherlink-mvp-is-now-available-on-tezos-ghostnet/5850">on Tezos
Ghostnet</a>
as it gets ready for Mainnet deployment.</li>
</ul>
<p>In 2024, we look forward to continuing to work closely with the Tezos
community to cement Tezos’ position as a technical leader in the
blockchain space. Stay tuned for more updates!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>In a recent research
<a href="https://eprint.iacr.org/2023/977.pdf">paper</a>, we described two
possible alternative designs we considered. The Oxford 2 protocol
proposal implements the algorithm presented in part 4.2. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Introducing Teztale – a Dashboard for Tezos Consensus2023-10-31T17:00:00+01:002023-10-31T17:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2023-10-31:/introducing-teztale.html<p>Monitor the Tezos consensus algorithm and analyze baker performance.</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: <a href="https://nomadic-labs.gitlab.io/teztale-dataviz">Teztale</a> is
a new consensus-inspection tool for Tezos, developed by Nomadic
Labs. This post shows you how to use it.</strong></p>
<p>Consensus algorithms play a central role in maintaining the integrity
and security of blockchain networks. They define the rules for how
nodes reach agreement on the state of the network and how new valid
blocks are added.</p>
<p>However, due to changing network conditions, it is not always a
straightforward process. To monitor the behavior of Tezos’ consensus
algorithm, Tenderbake, we have developed a monitoring tool called
Teztale. Initially built for our Incident Response Team, we are now
making Teztale available for anyone to use.</p>
<p><strong>Tip:</strong> If you are not already familiar with Tenderbake, we suggest
first taking a look at the following resources:</p>
<ul>
<li><a href="https://tezos.gitlab.io/active/consensus.html">Tezos’ consensus
algorithm</a> (documentation)</li>
<li><a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">A look ahead to
Tenderbake</a>
(blog post)</li>
<li><a href="https://research-development.nomadic-labs.com/tenderbake-has-been-injected.html">Tenderbake has been
injected</a>
(blog post)</li>
<li><a href="https://functori.com/blog/blog-tenderbake-baker-as-state-machine.html">Tenderbake’s Baker as a
StateMachine</a>
(blog post)</li>
<li><a href="https://arxiv.org/abs/2001.11965">A Solution to Dynamic Repeated Consensus for
Blockchains</a> (research article)</li>
</ul>
<h2 id="tenderbake-rounds"><a class="toclink" href="#tenderbake-rounds">Tenderbake Rounds</a></h2>
<p>In Tenderbake, a block level (also known as “block height”) is made up
of rounds, starting with round 0. Each round consists of three
successive phases:</p>
<ol>
<li>A <strong>proposal</strong> phase where one of the bakers proposes a new block
<em>payload</em> – the non-consensus content of the block (transfers,
smart contract calls, voting operations, etc.).</li>
<li>A <strong>pre-attestation</strong> voting phase, where validators vote to accept
the proposed payload.</li>
<li>An <strong>attestation</strong> voting phase, where validators vote upon the
contents of the whole candidate block.</li>
</ol>
<p>A <em>quorum</em> of votes (amounting to <a href="https://tezos.gitlab.io/active/consensus.html#validator-selection-staking-balance-active-stake-and-frozen-deposits">over 2/3 of the total active
stake</a>)
must be gathered for the proposed block at the end of each voting step.</p>
<p>Ideally, whenever consensus is reached for a proposal at some level
(i.e., attestation quorum is observed), each participant adds the
proposed block to their local copy of the blockchain. Then a new
instance of the algorithm is started for the next level.</p>
<p>However, things may go wrong for many reasons in practice. For
instance, messages such as block proposals or (pre-)attestations could
be lost or delayed, or some bakers could be desynchronized or
offline. Consequently, in some situations, participants may only reach
a consensus after multiple rounds.</p>
<p>Teztale enables you to keep track of this process as it unfolds. </p>
<h2 id="getting-the-data"><a class="toclink" href="#getting-the-data">Getting The Data</a></h2>
<p>At the top right corner of the <a href="https://nomadic-labs.gitlab.io/teztale-dataviz/">Teztale
page</a>, you see the
head of the chain (
<a href="https://nomadic-labs.gitlab.io/teztale-dataviz/Level#server=https%3A%2F%2Fteztale-mainnet.obs.nomadic-labs.cloud%2F&block=3797158&round=0&source=NL-mainnet-01-vigie&category=slots">#3797158</a>
in the example shown below) and the address of the Teztale server
providing data. If you use the data visualization page with your own
Teztale server, it can be <code>/</code> (the default value for data
visualization distribution), but any publicly accessible Teztale
server can be used. You can also use a Teztale server with a data
visualization page hosted somewhere else or running locally.</p>
<p>The Teztale <span class="caps">UI</span> gets data from a Teztale server connected to one or
more Teztale archivers that filter and forward data from a Tezos
node. The server gathers the data, deduplicates where needed, and
stores it in the database.</p>
<p>The server also serves the raw <span class="caps">JSON</span> data for a level through its
<em>/<<span class="caps">LEVEL</span>>.json</em> <span class="caps">API</span> endpoint. It contains basic information about
blocks concerned by a level (timestamp, validation/application time,
etc.), as well as information about each baker that should be involved
in consensus at this level. The information includes all consensus
operations published by the baker, when the consensus operations were
received by the archiver node, and whether they were included in a block.</p>
<p>It is possible to connect multiple nodes/archivers running on the same
Tezos network (Mainnet, testnets, local deployments) to a single
server. Aggregating data from multiple archivers makes the collected
information more reliable and representative, as it reduces biases
stemming from individual archivers’ performance. Moreover, it makes
the whole system more robust, ensuring that data is collected even if
one (or a few) archiver nodes are down.</p>
<p>The data visualization page can use multiple sources in different
ways, depending on what you are looking for: a single source, using
minimal time across all the sources, using the average value, etc.</p>
<h2 id="the-level-page"><a class="toclink" href="#the-level-page">The Level Page</a></h2>
<p><img alt="Image showing menu bar on Level page"
src="/images/teztale/menubar.png" width="100%"/></p>
<p>The navigation bar gives you quick access to different pages (Level,
Delay to Consensus, etc.).</p>
<!-- <img style="float: left; margin-right: 20pt" -->
<!-- alt="Image showing Round dropdown menu" -->
<!-- src="/images/teztale/rounds.png" width="20%"/> -->
<p>On the <a href="https://nomadic-labs.gitlab.io/teztale-dataviz/#server=https%253A%252F%252Fteztale-mainnet.obs.nomadic-labs.cloud%252F&round=0">Level
page</a>,
users can monitor events in real-time and investigate past behavior
based on various parameters. It has a sub-menu with a few options:</p>
<ul>
<li>
<p><strong>Level</strong> is the block level. By default it’s locked on the alias
Head-1, which means you are in ‘streaming mode’ and the page will
automatically refresh when a new block is added to the
chain. Pressing the pause button will halt the stream and let you
see the relevant block number (#3797157 in our case). You can also
enter any block level you would like to inspect. The arrow buttons
let you browse through block levels.</p>
</li>
<li>
<p><strong>Round</strong> allows you to select the round for a given level. When the
network runs smoothly, there is usually no more than one round
(round 0/0) in a block. For blocks where it has taken several rounds
to reach consensus, you can switch back and forth between rounds as
you would with level.</p>
</li>
<li>
<p><strong>Source </strong>is the Teztale archiver used as a data source.</p>
</li>
</ul>
<p>In order to see Teztale at work, we can use the extreme case of block
level
<a href="https://nomadic-labs.gitlab.io/teztale-dataviz/Level#server=https%3A%2F%2Fteztale.tzalpha.net%2F&block=3019851&round=0">#3019851</a>
as an example. This level took an exceptional 18 rounds for the
network to reach consensus (see <a href="https://forum.tezosagora.org/t/incident-report-slow-consensus-on-block-3019851/4940">our
investigation</a>),
causing approximately <em>52</em> minutes to elapse between block levels
<a href="https://nomadic-labs.gitlab.io/teztale-dataviz/Level#server=https%3A%2F%2Fteztale.tzalpha.net%2F&block=3019850&round=0">#3019850</a>
and
<a href="https://nomadic-labs.gitlab.io/teztale-dataviz/Level#server=https%3A%2F%2Fteztale.tzalpha.net%2F&block=3019852&round=0">#3019852</a>.</p>
<p>For blocks levels that were completed in a single round (normal
conditions), the Level page shows three sections:</p>
<ul>
<li>a histogram;</li>
<li>key block metrics and other info; and,</li>
<li>a table classifying consensus operations according to different
types (i.e., missed operations, valid operations, lost operations, etc.)</li>
</ul>
<p><img alt="Image showing Histogram on Level page"
src="/images/teztale/histogram.png"
width="100%"/></p>
<p>If you are inspecting a block level that has more than one round, as
it is the case with
<a href="https://forum.tezosagora.org/t/incident-report-slow-consensus-on-block-3019851/4940">#3019851</a>,
you will see an additional table classifying delegates <strong>according to
arrivals and departures</strong> (more on that below).</p>
<h2 id="the-histogram"><a class="toclink" href="#the-histogram">The Histogram</a></h2>
<p>The histogram tracks the reception of the block proposal for each
round, and its associated pre-attestations and attestations. The
x-axis tracks the elapsed time since round rights were enabled, and
the y-axis plots the delegates seen (pre-)attesting this proposal. At
the top right, you can choose whether the y-axis should display the
number of delegates or the number of slots.</p>
<p>The histogram also features additional indicators such as block
validation, block application, pre-quorum, and quorum.</p>
<p>You can also see a delegate’s individual performance for
(pre)attestation using the ‘Search delegate’ bar in the top right-hand
corner. Paste a baker’s tz-address and search, or select a baker in
the dropdown menu. The pre-attestation of the delegates you have
selected will be shown on the<a href="https://dictionary.cambridge.org/dictionary/english/map">
</a>histogram as
violet and green dotted<a href="https://dictionary.cambridge.org/dictionary/english/red">
</a>lines.</p>
<ul>
<li><strong>Pro tip 1 :</strong> Click and drag to zoom in on an area of the chart.</li>
<li><strong>Pro tip 2 :</strong> Click a bar to see (pre–)attestations details. From
there, you can either download a .csv file or copy the whole section.</li>
</ul>
<p><img alt="GIF showing Histogram on Level page"
src="/images/teztale/histogramgif.gif"
width="100%"/></p>
<p>Next to the histogram you’ll find information about the current block
and round. For example, you can see if a round is a reproposal of a
previous round. You can also find information about the preceding and
succeeding block.</p>
<h2 id="the-tables"><a class="toclink" href="#the-tables">The Tables</a></h2>
<p>The first table classifies the delegates’ consensus operations
according to the following scenarios:</p>
<ul>
<li><strong>Missed</strong>: These delegates were expected to participate in this
round, but the node failed to receive their attestations, which were
not included in the block.</li>
<li><strong>Valid</strong>: The node successfully received these delegates’
attestations, which were included in the block.</li>
<li><strong>Lost</strong>: The node successfully received these delegates’
attestations, but their attestations were not included in the block.</li>
<li><strong>Held Captive</strong>: Attestations from these delegates were included in
the block, but the archiver node did not receive these operations.</li>
<li><strong>Erroneous</strong>: A Teztale archiver reported something wrong with these delegates’ attestations.</li>
</ul>
<p>If the block level has more than one round, an additional table called
‘Delegates’ will be displayed. ‘Arriving’ delegates are validators
that joined in during the current round, while ‘exiting’ delegates are
validators that left during the current round.</p>
<p>Clicking a delegate’s address will present you with two options:
copying the delegate’s address, or going to the Delegate Stats page,
where you can take a closer look at any delegate’s performance.</p>
<h2 id="the-delegates-stats-page"><a class="toclink" href="#the-delegates-stats-page">The Delegates Stats page</a></h2>
<p>The Delegate Stats page comes in handy when you need to pinpoint if
and why a baker or the whole network is in trouble. Select the range
of blocks you want to review, select a delegate, and then choose
whether you want to display the delay or deviation (from the mean of
all delegates).</p>
<p>The example below shows <a href="https://nomadic-labs.gitlab.io/teztale-dataviz/DelegateStats#server=https%253A%252F%252Fteztale-mainnet.obs.nomadic-labs.cloud%252F&rangeLowerBound=-100&rangeUpperBound=4103170&delegate=tz1dbfppLAAxXZNtf2SDps7rch3qfUznKSoK">a baker missing a few attestations in a
row</a>.</p>
<p><img alt="Image showing Delegates Stats page"
src="/images/teztale/delegatesstats.png"
width="100%"/></p>
<p>At the left of the scatter plot, we can see that the baker’s
attestations were all received around 12 seconds after the candidate
block. This is late considering that since the activation of <a href="https://research-development.nomadic-labs.com/Mumbai-is-live.html">the
Mumbai
protocol</a>
on Tezos Mainnet, the minimal (and desired) block time is 15 seconds.</p>
<p>There are a number of possible causes for something like this to
happen, but the baker was able to identify and fix the problem quickly
(starting from block
<a href="https://nomadic-labs.gitlab.io/teztale-dataviz/Level#server=https%3A%2F%2Fteztale-mainnet.obs.nomadic-labs.cloud%2F&block=4103106&round=0">#4103106</a>),
as they knew there had been a problem with their clock not being on time.</p>
<h2 id="happy-monitoring"><a class="toclink" href="#happy-monitoring">Happy monitoring!</a></h2>
<p>Teztale was designed for our Incident Response Team as a tool to
monitor the overall health of Tezos network, but as we have shown in
this guide, Teztale’s utility is much broader.</p>
<p>Whether you are looking to get a better understanding of Tezos’
consensus algorithm at work, or need to inspect and analyze a specific
baker’s performance, Teztale has you covered.</p>Baking and the Oxford Proposal: A Technical Guide2023-09-21T18:00:00+02:002023-09-21T18:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2023-09-21:/prepare-for-oxford.html<p>New features change the workflow of bakers. Here is what you need to know.</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: The Oxford upgrade proposal contains changes that affect the workflow of bakers. This blog post covers what to be aware of if Oxford is activated.</strong></p>
<p>The <a href="https://research-development.nomadic-labs.com/oxford-announcement.html">Oxford protocol proposal</a> currently going through Tezos’ on-chain governance process contains several changes that affect not just staking economics, but also how bakers operate. </p>
<p>The most significant changes are subject to a separate vote and will only be activated later, provided they are accepted. However, other changes affecting bakers will take effect immediately after activation, if the Oxford protocol is adopted.</p>
<p>In this blog post we highlight what Tezos bakers should be aware of, and the necessary steps to take for ensuring a smooth transition.</p>
<h2 id="adaptive-issuance-and-the-new-staker-role"><a class="toclink" href="#adaptive-issuance-and-the-new-staker-role">Adaptive Issuance and the new <em>staker</em> role</a></h2>
<p>Adaptive Issuance is a new approach to staking economics on Tezos, adapting the network’s emission of its native token, tez, to fit better with real-world usage. For a deeper dive we recommend our blog post introducing <a href="https://research-development.nomadic-labs.com/adaptive-issuance.html">Adaptive Issuance</a>.</p>
<p>Adaptive Issuance will not be automatically enabled upon activation of the Oxford protocol. Instead, bakers will be able to vote using a configuration option when running the baking software. This process is described in detail further below.</p>
<p>Adaptive Issuance also <a href="https://research-development.nomadic-labs.com/oxford-announcement.html#new-staking-mechanism">introduces the role of staker</a>, in addition to the existing roles of baker (or delegate) and delegator. However, no account can become a staker unless Adaptive Issuance is voted in by the additional vote.</p>
<h2 id="changes-in-a-bakers-workflow"><a class="toclink" href="#changes-in-a-bakers-workflow">Changes in a baker’s workflow</a></h2>
<p>In order to ease the potential transition to Adaptive Issuance, additional changes have been included in the Oxford protocol proposal<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>, which will take effect immediately from the protocol proposal’s activation.</p>
<p>They do not affect baking/attestation rights or rewards, but they affect the actions required to operate a baker. These changes are:</p>
<ul>
<li><strong>No more automatic freezing and unfreezing.</strong> Currently, the protocol regularly adjusts the baker’s frozen balance to accommodate for variations in their delegated balance. With Oxford, bakers may adjust their staked balance (which is frozen) via new explicit “stake” and “unstake” operations. This gives them more control, as the protocol will never move tez from their spendable balance without the baker explicitly signing an operation for it.</li>
<li><strong>Changes to slashing.</strong> Double-baking penalties are changed from a fixed sum of 640 tez to 10% of the staked balance<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup>. Denunciation rewards (for double-baking and double-attestations) are reduced from one half to one-seventh of the slashed funds.</li>
</ul>
<p>We also remind bakers that the “endorsement” and “preendorsement” operations are renamed “attestation” and “preattestation”, respectively. This can affect <span class="caps">API</span> calls or automated log review setups.The renaming is due to the fact that a baker does not actually endorse a block as much as attest to its existence and validity, which this new terminology makes clear. This change was initiated in Nairobi and is completed with Oxford. </p>
<h2 id="transitioning-from-nairobi-to-oxford"><a class="toclink" href="#transitioning-from-nairobi-to-oxford">Transitioning from Nairobi to Oxford</a></h2>
<p>A smooth transition to Oxford requires bakers to take steps which they are familiar with from previous protocol upgrades. In this case:</p>
<ul>
<li>Update to <a href="https://forum.tezosagora.org/t/announcing-octez-v18-0/5746">Octez
v18</a>.</li>
<li>Run both Nairobi and Oxford bakers (and accusers) in parallel at the time of activation. The former will automatically stop participating in consensus, and the latter will take over.</li>
</ul>
<p>There is no requirement for bakers to take further immediate action in order to continue baking after activation, but some changes need to be taken into consideration for optimal operation of especially public bakers in the longer term.</p>
<p><strong>Frozen deposits become stakes</strong></p>
<p>Upon activation of Oxford, the frozen deposit of existing bakers will be preserved and the same amount will become the baker’s stake.</p>
<p>The baking and attesting rights are calculated in a similar fashion as before:</p>
<ul>
<li>Rights are proportional to the baker’s delegated balance, that is, the baker’s own balance plus the balance of all their delegators.</li>
<li>Rights from delegations are capped at 9x the baker’s own staked balance. Above that, the baker is <em>overdelegated,</em> and any additional delegation does not bring additional rights or rewards.</li>
</ul>
<p>However, while previously the frozen deposits would automatically adjust to the delegate’s total balance, in Oxford the staked balance does not. This will require manual adjustments.</p>
<p><strong>How to manage stake</strong></p>
<p>Oxford introduces a series of new commands for managing a baker’s staked balance.</p>
<p>Adding tez to the stakebalance is done by executing this <span class="caps">CLI</span> command:</p>
<div class="highlight"><pre><span></span><code><span class="nv">octez</span><span class="o">-</span><span class="nv">client</span><span class="w"> </span><span class="nv">stake</span><span class="w"> </span><span class="mi">2000</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="o"><</span><span class="nv">baker_key</span><span class="o">></span>
</code></pre></div>
<p>This will make 2000 of the baker’s spendable tez immediately unavailable for transfer, and the amount will be included in the computation of baking rights after 6 cycles.</p>
<p>Reducing the staked balance is done with this command:</p>
<div class="highlight"><pre><span></span><code><span class="nv">octez</span><span class="o">-</span><span class="nv">client</span><span class="w"> </span><span class="nv">unstake</span><span class="w"> </span><span class="mi">2000</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="o"><</span><span class="nv">baker_key</span><span class="o">></span>
</code></pre></div>
<p>If a baker wishes to stop baking, they may bring their stake to zero:</p>
<div class="highlight"><pre><span></span><code><span class="nv">octez</span><span class="o">-</span><span class="nv">client</span><span class="w"> </span><span class="nv">unstake</span><span class="w"> </span><span class="nv">everything</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="o"><</span><span class="nv">baker_key</span><span class="o">></span>
</code></pre></div>
<p>After executing unstake commands, the baker must wait for an unfreezing period of 7 cycles. During this time, the amount in question is no longer taken into account to determine baking rights, but is still subject to slashing. Afterwards, they need to <em>finalize their unstake operation</em> with the following command:</p>
<div class="highlight"><pre><span></span><code><span class="nv">octez</span><span class="o">-</span><span class="nv">client</span><span class="w"> </span><span class="nv">finalize</span><span class="w"> </span><span class="nv">unstake</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="o"><</span><span class="nv">baker_key</span><span class="o">></span>
</code></pre></div>
<p>Note that these operations are <a href="https://tezos.gitlab.io/oxford/adaptive_issuance.html?highlight=pseudo#new-staking-mechanism">pseudo-operations</a>: a transfer operation, but where the destination matches the source. Therefore <em>they must be signed by the baker’s manager key</em> and not the <a href="https://tezos.gitlab.io/user/key-management.html#consensus-key">consensus key</a>.</p>
<p><strong>New bakers must explicitly stake</strong></p>
<p>Up until now, an account could become a baker by using just this command:</p>
<div class="highlight"><pre><span></span><code><span class="n">octez</span><span class="o">-</span><span class="n">client</span><span class="w"> </span><span class="n">register</span><span class="w"> </span><span class="n">key</span><span class="w"> </span><span class="o"><</span><span class="n">baker_key</span><span class="o">></span><span class="w"> </span><span class="n">as</span><span class="w"> </span><span class="n">delegate</span>
</code></pre></div>
<p>Under Oxford, it is no longer sufficient just to register as a delegate to start baking. It results in no tez being added to the staked balance and hence no baking rights being assigned. </p>
<p>The baker needs to subsequently run the stake command in order to stake at least 6,000 tez, before they will have rights assigned to them:</p>
<div class="highlight"><pre><span></span><code><span class="nv">octez</span><span class="o">-</span><span class="nv">client</span><span class="w"> </span><span class="nv">stake</span><span class="w"> </span><span class="mi">6000</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="o"><</span><span class="nv">baker_key</span><span class="o">></span>
</code></pre></div>
<p><strong>Set deposits limit operation is removed</strong></p>
<p>Prior to Oxford, baking rewards have accrued to the spendable balance. But at the end of every cycle, the protocol automatically increases or decreases the frozen deposit based on the variations of the delegated balance. </p>
<p>To avoid public bakers risking all of their balance being frozen, with no tez left to pay their delegators, it has been possible to set an upper bound for the frozen balance. This is known as the <a href="https://tezos.gitlab.io/nairobi/consensus.html#validator-selection-staking-balance-active-stake-and-frozen-deposits">deposit limit</a>.</p>
<p>In Oxford, the staked balance is always set explicitly, and therefore the “set deposits limit” operation is deprecated.</p>
<h2 id="adjusting-stake-as-balance-changes"><a class="toclink" href="#adjusting-stake-as-balance-changes">Adjusting stake as balance changes</a></h2>
<p>Baking requires at least 10% of the total balance staked to maximize rewards. But as the Oxford protocol does not freeze/unfreeze tez automatically, it is the responsibility of the baker to adjust as needed.</p>
<p>As baking/attestation rewards are received, the total balance increases over time. Therefore, it is advisable for a baker without delegators to have a staked balance slightly above 10%, and to adjust it upwards from time to time<sup id="fnref:3"><a class="footnote-ref" href="#fn:3">3</a></sup>. </p>
<p>Note that if Adaptive Issuance is later activated, going above 10% staked will increase rewards, due to staked funds counting twice as much as (self)delegated funds in calculating baking rights and voting power. Hence, staking 100% will generate more rewards than 10%. But until Adaptive Issuance activates, going above 10% will only increase funds at risk of slashing without increasing rewards.</p>
<p><strong>Bakers accepting delegations</strong> will want to set their staked balance high enough to leave room for new delegations, while still keeping it below their total balance, so they are able to pay out rewards. A baker can accept up to 9x their staked balance before being overdelegated.</p>
<h2 id="activation-of-adaptive-issuance-and-new-staking-mechanism"><a class="toclink" href="#activation-of-adaptive-issuance-and-new-staking-mechanism">Activation of Adaptive Issuance and new staking mechanism</a></h2>
<p>The activation of Adaptive Issuance and the new _staker _role is decided in a separate signaling vote, which begins if and when the Oxford proposal is activated. </p>
<p>Bakers will be able to vote ‘On’ (in favour), ‘Off’ (against), or ‘Pass’. Abstaining from explicit signaling will have the same effect as a ‘Pass’ vote.</p>
<p>Before voting, bakers are invited to first familiarize themselves with <a href="https://research-development.nomadic-labs.com/adaptive-issuance.html">Adaptive Issuance</a>. Should they want to vote in favor of its activation, they must pass the following parameter when running their Oxford baker:</p>
<div class="highlight"><pre><span></span><code>—-adaptive-issuance-vote on
</code></pre></div>
<p>To vote against activation:</p>
<div class="highlight"><pre><span></span><code>—-adaptive-issuance-vote off
</code></pre></div>
<p>To explicitly vote pass:</p>
<div class="highlight"><pre><span></span><code>—-adaptive-issuance-vote pass
</code></pre></div>
<p>The signaling vote is driven by an Exponential Moving Average (<span class="caps">EMA</span>) whose half-life is 2 weeks. That is, it takes two weeks for the <span class="caps">EMA</span> to raise from 0% to 50% assuming that <em>only</em> On votes are cast. The target threshold is a supermajority of 80% ‘On’ votes out of all ‘On’ and ‘Off’ votes. </p>
<p>The signaling vote has no time limit and continues to run, as long as the threshold is not met. There is also no quorum. Lack of participation (equal to ‘Pass’ votes) is not taken into account by the <span class="caps">EMA</span>, and hence only affects the time required to reach the threshold<sup id="fnref:4"><a class="footnote-ref" href="#fn:4">4</a></sup>.</p>
<p>If the threshold is met, the Voting phase will complete at the end of the current cycle, and an Adoption phase lasting 7 cycles (about 20 days) will begin.</p>
<h2 id="oxford-on-test-networks"><a class="toclink" href="#oxford-on-test-networks">Oxford on test networks</a></h2>
<p>The protocol-specific ‘<a href="https://teztnets.xyz/oxfordnet-about">Oxfordnet</a>’ test network is now active and available for you to test the new <span class="caps">CLI</span> commands. Adaptive Issuance has been enabled on this network, so the distribution of baking rights and reward amounts will differ from the focus of this blog post: the time period where Oxford protocol is activated, but where Adaptive Issuance is not activated.</p>
<p>Should Oxford be voted in during the Promotion Period, the permanent Tezos test network, <a href="https://teztnets.xyz/ghostnet-about">Ghostnet</a>, will upgrade to Oxford one to two weeks before mainnet. Adaptive Issuance will be off on Ghostnet until voted in by Ghostnet bakers. Ghostnet bakers are recommended to signal vote for Adaptive Issuance being kept ‘Off’ for the time being, so as to mimic mainnet.</p>
<h2 id="questions"><a class="toclink" href="#questions">Questions?</a></h2>
<p>The Oxford protocol proposal represents a major change to Tezos staking, and it’s natural that questions arise. We invite you to ask questions on the <a href="https://tezos.stackexchange.com/questions/tagged/adaptive-issuance">Tezos Stack Exchange</a>, and label them with the tag ‘adaptive-issuance’. This way, we will build a community-driven, collaborative knowledge base on the topic for all to benefit from.</p>
<p>For further information on Adaptive Issuance and other changes brought about by the Oxford protocol proposal, consider the following resources:</p>
<p><a href="https://research-development.nomadic-labs.com/oxford-announcement.html">Announcing Oxford, Tezos’ 15th protocol upgrade proposal</a> (Nomadic Labs)</p>
<p><a href="https://research-development.nomadic-labs.com/adaptive-issuance.html">Adaptive Issuance and Staking</a> (Nomadic Labs)</p>
<p><a href="https://medium.com/the-aleph/a-walkthrough-of-tezos-new-staking-mechanism-4ff0c50a57a8">A Walkthrough of Tezos’ New Staking Mechanism</a> (Nicolas Ochem / Oxhead Alpha)</p>
<p><a href="https://spotlight.tezos.com/understanding-adaptive-issuance-in-less-than-10-minutes/">Understanding “Adaptive Issuance” In Less Than 10 Minutes</a> (Cryptonio.tez / Tezos Commons)</p>
<p><em>Special thanks to Nicolas Ochem for his contributions to this blog post.</em></p>
<!-- Footnotes themselves at the bottom. -->
<h2 id="notes"><a class="toclink" href="#notes">Notes</a></h2>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>With these upfront changes, the potential later transition to Adaptive Issuance only consists of a change to new distribution and reward formulas. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>To leave room for new delegations, public bakers must maintain a larger staked balance than what their own total stake requires. However, this increases their slashing risk, and this parameter should be adjusted carefully. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>Given the max network issuance of 5% annually, setting staked balance to 12.1% of current total balance is enough to ensure that you are still above 10% after one year. Note that this only applies to bakers not accepting delegations, and assumes no further funds are added to the total balance. <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:4">
<p>This design of the voting mechanism allows time for discussions and possibly other amendments. If Oxford is activated and all bakers vote in favor immediately, it would still take ~33 days to reach the 80% threshold, and then a further ~20 days to activate the features. Any Pass or No votes would extend this period. <a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
</ol>
</div>We’re doing 1 million TPS on Tezos! Here’s how2023-08-24T14:00:00+02:002023-08-24T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2023-08-24:/million-tps-demo.html<p>Unleashing the power of Smart Rollups in a public demonstration.</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: In the spirit of “show, don’t tell”, we are proud to present a demonstration of Tezos’ ability to process one million <span class="caps">TPS</span>, using Smart Rollups and a data-availability solution built for Tezos.</strong></p>
<p>With Smart Rollups <a href="https://research-development.nomadic-labs.com/Mumbai-is-live.html">live on Mainnet</a> since March, it’s time to demonstrate the power of this technology to the world!</p>
<p>Smart Rollups represent a groundbreaking design philosophy in that they enable custom rollup solutions to be built using any programming language that compiles to <span class="caps">WASM</span>. See <a href="https://tezos.com/developers/smart-rollups/">this page</a> for a quick introduction, or the <a href="https://tezos.gitlab.io/alpha/smart_rollups.html">Tezos documentation</a> for a more technical walk-through.</p>
<p>But Smart Rollups also play another crucial role: scaling the Tezos network to serve billions of users engaging in high-throughput activities, enabling gas intensive computation, and keeping fees low.</p>
<p>This is done with a combination of horizontal and vertical scaling, as described in our <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html#the-best-of-both-worlds">blog post on how to scale Tezos</a> from early 2022. Now we are happy to be able to show the outlined approach in action, with a public demo showcasing a throughput of 1 million transactions per second (<span class="caps">TPS</span>).</p>
<h2 id="watch-each-transaction-happen"><a class="toclink" href="#watch-each-transaction-happen">Watch each transaction happen</a></h2>
<p>To show the throughput achieved in the demo as more than a few metrics on a dashboard, the setup includes a visual representation of the transactions as they happen.</p>
<p>The visualization consists of a series of 5-megapixel images being displayed one by one on a webpage. </p>
<p>Each transaction in the demo contains information about one <span class="caps">RGB</span> color component in one pixel, meaning that three transactions are required to update a pixel. Hence, the display of each 5-megapixel image represents 15 million transactions.</p>
<p>As the transactions are executed, the image changes progressively, showing a full change of image for each 15-second block. 15 million transactions executed in each 15-second block: <em>one million transactions per second</em>.</p>
<p>The demo runs for two minutes, and a total of eight images are displayed. We should perhaps emphasize that displaying these images serves purely to create a more tangible experience of the throughput achieved. The transactions can represent any type of blockchain operation.</p>
<p>The visualization setup was created by Elevated Labs, and the technical details were presented at the recent <a href="https://tezos.com/tez-dev/">TezDev developer conference</a> in Paris. Here, Trilitech’s Emma Turner and Nomadic Labs’ Thomas Letan also gave the audience an exclusive preview of the demo. Watch their presentation and the demo preview <a href="https://www.youtube.com/watch?v=2EgjMvEIGww">here</a>.</p>
<!--
<iframe width="930" height="523" src="https://www.youtube.com/embed/2EgjMvEIGww?si=aY9HvhC3eKX4y_t-" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
-->
<h2 id="a-realistic-demonstration"><a class="toclink" href="#a-realistic-demonstration">A realistic demonstration</a></h2>
<p>The demo is designed to be as realistic and publicly verifiable as possible. In particular, we are not cutting corners when it comes to defining a transaction.</p>
<ul>
<li>Transactions in this demo consist of the transfer of a <a href="https://tezos.gitlab.io/alpha/tickets.html">Tezos ticket</a> between two rollup accounts.</li>
<li>Transaction size is similar to that of transactions typically processed on Mainnet<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>. </li>
<li>Every transaction is individually signed and checked. It’s an important factor when comparing <span class="caps">TPS</span> claims across different blockchains, because these are the costly aspects of the computation done for each transfer.</li>
<li>Rollups are working exactly as they would in a real-world setting (publishing commitments, etc.).</li>
<li>The procedure and code will be made public, so that any third-party may reproduce it.</li>
</ul>
<p>The demo is carried out on the public testnet <a href="https://teztnets.xyz/mondaynet-about">Mondaynet</a> with 1,000 Smart Rollup nodes (horizontal scaling) each processing 1000 <span class="caps">TPS</span> (vertical scaling). </p>
<p>All nodes are deployed in Google Cloud on 600 ‘<a href="https://cloud.google.com/compute/docs/compute-optimized-machines">c2-standard-16</a>’ instances. The transactions themselves could be executed with half the computing power, but the extra <span class="caps">CPU</span> is required for the visualization. </p>
<h2 id="avoiding-data-bottlenecks-with-dacs"><a class="toclink" href="#avoiding-data-bottlenecks-with-dacs">Avoiding data bottlenecks with DACs</a></h2>
<p>In total, transactions equivalent to about 16 gigabytes of data are carried out over the two minutes, requiring a bandwidth of about 133 megabytes/second.</p>
<p>This is far above Tezos’ current block capacity. A protocol-level solution to this bandwidth challenge, the <a href="https://research-development.nomadic-labs.com/data-availability-layer-tezos.html">Data-Availability Layer</a>, is currently under development and is expected to launch in early 2024.</p>
<p>In the meantime, to avoid making Layer 1 a bottleneck in this demo, we use Data Availability Committees (DACs), with each rollup having its own <span class="caps">DAC</span>, i.e. 1000 instances. For a deeper dive into DACs, <a href="https://research-development.nomadic-labs.com/introducing-data-availability-committees.html">see this article</a>. </p>
<p>The principle is this: Instead of posting newly received rollup messages directly to Layer 1 in their raw form, the messages are given to a <span class="caps">DAC</span>, which stores the data off-chain, creates a Merkle tree with hashes of all the messages, and returns a root hash signed by the <span class="caps">DAC</span>. Together, the root hash and signature(s) are called a <em><span class="caps">DAC</span> certificate</em>.</p>
<p>The <span class="caps">DAC</span> certificate is then posted to the rollup inbox on Layer 1. Anyone running a Smart Rollup node can verify the contents covered by the root hash by requesting the input data from the <span class="caps">DAC</span> using the <a href="https://tezos.gitlab.io/alpha/smart_rollups.html#reveal-data-channel">reveal data channel</a>.</p>
<p>The use of DACs makes the setup very efficient in terms of bandwidth, as it allows us to represent the full 16 gigabytes of data in just 8000 Layer 1 messages (~1.2 megabytes in total), with each of the 1,000 <span class="caps">DAC</span> instances posting one message in each of the eight blocks that are produced during the 2-minute demo. This is far from the 120 million messages it would require to post all transactions directly to Layer 1.</p>
<h2 id="were-not-stopping-at-one-million"><a class="toclink" href="#were-not-stopping-at-one-million">We’re not stopping at one million</a></h2>
<p>Reaching a million <span class="caps">TPS</span> is just a first milestone, as the work to scale Tezos continues.</p>
<p>In a keynote presentation at TezDev 2023 entitled ‘<a href="https://www.youtube.com/watch?v=6USr8iqXXFE">Tezos 2.0: The Next Era of Rollups - Ultra High Throughput</a>’, Tezos co-founder Arthur Breitman outlined a way to scale Tezos even further with a rollup-centric approach. Other talks and highlights from the TezDev conference are available <a href="https://www.youtube.com/@Tezos/videos">here</a>.</p>
<!--
<iframe width="100%" src="https://www.youtube.com/embed/6USr8iqXXFE?si=U5LQnYptPdKc04A9" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
-->
<p>We take pride in making this demo realistic and reproducible. Should you wish to check our work and perhaps set up your own test, we are happy to assist.</p>
<p>And similarly, don’t hesitate to <a href="https://adoption-support.nomadic-labs.com/contact/">reach out</a> if you are looking for a blockchain solution that is future-proof, flexible, highly scalable, and not least, fully decentralized. Because that is exactly what Smart Rollups are all about.</p>
<!-- Footnotes themselves at the bottom. -->
<h2 id="notes"><a class="toclink" href="#notes">Notes</a></h2>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>About 160 bytes. The payload includes sender and receiver addresses, an amount, the <span class="caps">RGB</span> component identifier (which can be seen as a token), and the transaction signature. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Announcing Oxford, Tezos’ 15th protocol upgrade proposal2023-08-11T15:00:00+02:002023-08-11T15:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2023-08-11:/oxford-announcement.html<p>Adaptive Issuance and Staking, new Timelocks and more!</p><p><em>This is a joint post from Nomadic Labs, Marigold, TriliTech, Oxhead
Alpha, Tarides, DaiLambda, <span class="amp">&</span> Functori.</em></p>
<p>Following the smooth activation of the <a href="https://research-development.nomadic-labs.com/Nairobi-is-live.html">Nairobi protocol
upgrade</a>
on June 24 — and a successful <a href="https://tezos.com/tez-dev/">TezDev</a>
conference on July 21 — we are happy to unveil our latest Tezos
protocol proposal.</p>
<p>For our 15th upgrade proposal, we head to Oxford, home to the world’s
second-oldest university in continuous operation, with evidence of
teaching happening as early as 1096. As usual, the proposal’s “true”
name is its hash,
<code>ProxfordSW2S7fvchT1Zgj2avb5UES194neRyYVXoaDGvF9egt8</code>.</p>
<p>The Oxford proposal first and foremost contains two major changes to
the staking economics of Tezos. Note that these changes <em>will not be
automatically enabled upon activation,</em> if the Oxford proposal is
adopted. Instead, bakers will be able to signal their position in a
separate vote. The changes in question are:</p>
<ul>
<li><strong><a href="#adaptive-issuance-and-staking">Adaptive Issuance:</a></strong> a new
approach to tez issuance in the Tezos economic protocol, where
emission is adjusted dynamically, and tied to the ratio of staked
tez over the total supply.</li>
<li><strong><a href="#new-staking-mechanism">A new Staking mechanism:</a></strong> a reworking
of PoS on Tezos, which introduces the new role of <em>staker</em> in
addition to the existing <em>delegate</em> (also known as <em>baker</em>) and
<em>delegator</em> roles.</li>
</ul>
<p>In this post we provide an overview of the functionality. For a deeper
and more technical description, see <a href="https://research-development.nomadic-labs.com/adaptive-issuance.html">the specification
document</a>.</p>
<p>Other prominent changes in Oxford (enabled upon activation):</p>
<ul>
<li><strong><a href="#refinements-to-pos-slashing-and-rewards">Refinements to PoS penalties and
rewards:</a></strong> slashing
penalties are made proportional to funds at stake, and new tools for
easier management of staked funds are introduced.</li>
<li><strong><a href="#heading=h.e220ily2luk4">Timelocks are back:</a></strong> a new design and
implementation of Timelocks has been finalized, addressing security
concerns that led to their temporary deactivation in a previous
protocol upgrade.</li>
<li><strong><a href="#timelocks-are-re-enabled">Further improvements of Smart
Rollups:</a></strong> new features simplify the
deployment of Smart Rollups both on public periodic test networks
and <em>ad-hoc</em> dedicated ones.</li>
</ul>
<p>The protocol proposal also includes other minor changes and
improvements which can be found in Oxford’s
<a href="https://tezos.gitlab.io/protocols/018_oxford.html">changelog</a>.</p>
<h2 id="adaptive-issuance-and-staking"><a class="toclink" href="#adaptive-issuance-and-staking">Adaptive Issuance and Staking</a></h2>
<p><a href="https://research-development.nomadic-labs.com/adaptive-issuance.html#adaptive-issuance">Adaptive
Issuance</a>,
originally
<a href="https://forum.tezosagora.org/t/adaptive-inflation/4552">introduced</a>
as “Adaptive Inflation”, is an evolution of the staking mechanism,
adapting the economics of tez to fit better with real-world usage, as
argued in the Tezos Agora post <a href="https://forum.tezosagora.org/t/adaptive-inflation/4552/95#why-adaptive-inflation-matters-for-tezos-1">“Why adaptive inflation matters for
Tezos”</a>.</p>
<p>The proposed mechanism ties the Tezos protocol’s regular issuance of
tez (from participation rewards and the Liquidity Baking subsidy), to
the ratio of staked tez over the total supply. At the end of each
blockchain cycle, the nominal issuance rate is recomputed to nudge the
staked funds ratio towards a protocol-defined target of 50%. When the
ratio of staked funds decreases and diverges from the target, emission
rates will increase, incentivizing participants to stake funds to
re-approach the target, and <em>vice versa</em>.</p>
<p>Consequently, the value of participation rewards and the Liquidity
Baking subsidy are no longer fixed values determined by protocol
constants – they <a href="https://research-development.nomadic-labs.com/adaptive-issuance.html#adaptive-rewards">rather change
automatically</a>
as the mechanism encourages (or discourages) staking of funds.</p>
<h3 id="new-staking-mechanism"><a class="toclink" href="#new-staking-mechanism">New Staking Mechanism</a></h3>
<p>A new role — <em>staker</em> — is introduced, in addition to <em>delegate</em> and
<em>delegator</em>. It enables tez holders to contribute to a baker’s
security deposit without the baker taking custody of their funds, with
in-protocol reward sharing.</p>
<p>When a <em>staker</em> provides <em>staking</em> funds, these funds are frozen and
are subject to both rewards and slashing, in proportion to their
weight in their delegate’s deposit. Stakers can modify or remove their
stakes, with the changes taking effect after 5 cycles for staking and
7 cycles for unstaking.</p>
<p><em>Delegates</em> (or <em>bakers</em>) can configure their staking policy by
setting parameters which explicitly state whether they accept staking
by stakers, and if so, up to which fraction of their total deposit. By
default, delegates do not accept any funds from stakers.</p>
<p>To encourage staking over liquid delegation, staked and delegated
funds have different weights in the computation of a delegate’s baking
and voting powers: staked funds (own and external) count twice as much
as delegated funds.</p>
<h3 id="refinements-to-pos-slashing-and-rewards"><a class="toclink" href="#refinements-to-pos-slashing-and-rewards">Refinements to PoS: slashing and rewards</a></h3>
<p>Additional changes to the staking mechanism and economic incentives
are also included in the Oxford protocol proposal. These are
independent of Adaptive Issuance and the new staking mechanism, and
would take effect from the protocol proposal’s activation:</p>
<ul>
<li>Delegates will have more control on their frozen deposits via a
dedicated interface. There will be<strong> no more automatic freezing and
unfreezing</strong> of deposits.</li>
<li>Double-baking penalties are changed from a fixed sum of 640 tez to
10% of the frozen deposit, making the slashed funds proportional to
the delegate’s (and potential stakers’) funds at stake.</li>
<li>Denunciation rewards (for double-baking and double-attestations) are
reduced from one half to one-seventh of the slashed funds.</li>
<li>Bakers can have a part of their rewards paid directly to their
frozen deposits, making management of staked funds easier.</li>
</ul>
<h3 id="activating-adaptive-issuance-and-staking"><a class="toclink" href="#activating-adaptive-issuance-and-staking">Activating Adaptive Issuance and Staking</a></h3>
<p>As previously mentioned, approval of the Oxford protocol proposal by
the community will not immediately enable Adaptive Issuance and the
new Staking mechanism upon protocol activation. Instead, it will
enable a <em>per-block vote</em>: a continuous signaling vote giving bakers
the opportunity to activate Adaptive Issuance and Staking on Tezos Mainnet.</p>
<p>Concretely, the features guarded behind the voting mechanism are:</p>
<ul>
<li>Adaptive Issuance (and adaptive rewards).</li>
<li>Ability for delegators to become stakers.</li>
<li>The changes in relative weights for staked and delegated funds for
the computation of a delegate’s baking and voting power.</li>
</ul>
<p>This choice is based on the following considerations:</p>
<ul>
<li>Separating the acceptance of Adaptive Issuance (and the rest of the
guarded features), from other contributions and changes included in
Oxford avoids the technical cost of creating, testing, and
maintaining two proposals: one with and one without Adaptive Issuance.</li>
<li>It gives the community more time to evaluate and discuss the
adoption of the feature, without blocking the protocol amendment process.</li>
</ul>
<p>The threshold for enabling the guarded features is set to 80% <strong>Yes</strong>
votes out of <strong>Yes + No</strong> votes — the same used for protocol
amendments — to ensure a high degree of community consensus. The vote
however differs from protocol amendment votes, in that the voting
phase is driven by an exponential moving average (<span class="caps">EMA</span>), and there is
no quorum. Absence of signaling will count as a <strong>Pass </strong>vote, which
is not taken into account by the <span class="caps">EMA</span>. If the threshold is reached, the
guarded features will be activated 7 cycles later.</p>
<p>A high-level functional specification of Adaptive Issuance and the
new Staking mechanism is given in: <a href="https://research-development.nomadic-labs.com/adaptive-issuance.html">“Adaptive Issuance and
Staking”</a>. For
more insight into the rationale and design choices behind these
features, see the <a href="https://forum.tezosagora.org/t/adaptive-inflation/4552">original
proposal</a> on
Tezos Agora and <a href="https://www.youtube.com/watch?v=3U8_L_1rY74&ab_channel=%23BlockchainEvolved">this
episode</a>
of the Blockchained Evolved Show by Tezos Commons.</p>
<h2 id="timelocks-are-re-enabled"><a class="toclink" href="#timelocks-are-re-enabled">Timelocks are re-enabled</a></h2>
<p>The Michelson op-codes enabling support for cryptographic
<a href="https://tezos.gitlab.io/oxford/timelock.html">Timelocks</a> are
re-enabled in Oxford.</p>
<p>Timelocks address a challenge with blockchain-based transactions,
primarily when trading, known as Maximal Extracted Value (<span class="caps">MEV</span>). Since
a transaction can be observed in the mempool before it is included in
a block, a user can exploit knowledge of the pending transaction to
their advantage against another user. For example, upon receiving a
transaction, a baker could craft a block including this transaction
and one of their own such that the sequential execution of these two
transactions guarantees a gain to the baker.</p>
<p>The use of Timelocks can mitigate (but not entirely prevent) this kind
of value extraction, by enabling the payload of a transaction to be
encrypted until it is too late to change the order of transactions.</p>
<p>Timelocks were disabled in the <a href="https://research-development.nomadic-labs.com/lima-is-live.html">Lima protocol
upgrade</a>,
following <a href="https://forum.tezosagora.org/t/vulnerability-found-in-the-timelock-feature/4660">the discovery of a security
flaw</a>
at the application level. The feature has since been redesigned, and
after thorough testing, including an external audit performed by
<a href="https://inference.ag/">Inference <span class="caps">AG</span></a>, we are ready to re-activate
this feature in the Tezos protocol.</p>
<p>Additionally, the new version of Timelock features new <a href="https://tezos.gitlab.io/oxford/cli-commands.html">client <span class="caps">CLI</span>
commands</a> allowing
users to create, open and verify time-locked chests, as well as to
execute the pre-computations required for fast chest generation.</p>
<h2 id="improvements-to-smart-rollups"><a class="toclink" href="#improvements-to-smart-rollups">Improvements to Smart Rollups</a></h2>
<p>As with the Nairobi protocol, the Oxford protocol proposal brings
several <a href="https://tezos.gitlab.io/protocols/018_oxford.html#smart-rollups"><em>quality of life
improvements</em></a>
for Smart Rollup developers:</p>
<ul>
<li>
<p>The rollup origination operation has been simplified. With Oxford,
it no longer requires a so-called <em>“origination proof”</em> as an
argument. This should benefit projects building on top of the Smart
Rollups infrastructure stack.</p>
</li>
<li>
<p>Similarly to what was already possible for smart contracts, it is
now possible to hard-code the origination of Smart Rollups in a
chain’s genesis block. This is used in the deployment of <a href="https://research-development.nomadic-labs.com/evm-tezos-testnet.html"><span class="caps">EVM</span> rollup
on test
networks</a>.</p>
</li>
<li>
<p>A new <span class="caps">WASM</span> <span class="caps">PVM</span> revision is released (<code>2.0.0-r2</code>). It enriches the
set of host functions available for querying the durable storage of
a Smart Rollup with a faster alternative to <code>store has</code>.</p>
</li>
</ul>
<h2 id="tezos-evolves-again"><a class="toclink" href="#tezos-evolves-again">Tezos evolves, again</a></h2>
<p>We consider the new staking economics proposed in Oxford another good
example of Tezos’ ability to evolve and adapt, and look forward to the
community discussions on the activation of the Adaptive Issuance and
Staking features.</p>
<p>Note that if this proposal is voted in by the community, upgrading to
Octez v18.0 (or later) will be necessary for participating in consensus.</p>
<p>In order to allow the community to start testing the Oxford proposal
as soon as possible, <code>v18.0~rc1</code>, a release candidate for Octez v18.0,
will be published in the coming days. A dedicated protocol test
network, Oxfordnet, is also scheduled to launch soon after.</p>Adaptive Issuance and Staking2023-08-11T14:00:00+02:002023-08-11T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2023-08-11:/adaptive-issuance.html<p>This document provides a high-level functional specification for Adaptive Issuance and Staking, two new features of the Oxford protocol upgrade proposal, which together constitute a major evolution of Tezos’ Proof-of-Stake mechanism.
A high-level functional specification for Adaptive Issuance and Staking, a major evolution of Tezos’ Proof-of-Stake.</p><p><span class="math">\(\newcommand\F[2]{\mathit{#1}\left(#2\right)}\)</span>
<span class="math">\(\newcommand{\minR}{\mathit{min_r}}\)</span>
<span class="math">\(\newcommand{\maxR}{\mathit{max_r}}\)</span>
<span class="math">\(\newcommand{\tmult}{\cdot}\)</span>
<span class="math">\(\newcommand\static[1]{\F{static}{#1}}\)</span>
<span class="math">\(\newcommand{\sfr}{\frac{1}{1600}}\)</span>
<span class="math">\(\newcommand\tc{\tau_c}\)</span>
<span class="math">\(\newcommand\tr{\tau_r}\)</span>
<span class="math">\(\newcommand\grf{\gamma}\)</span>
<span class="math">\(\newcommand\dyn[1]{\F{dyn}{#1}}\)</span>
<span class="math">\(\newcommand\sgn[1]{\F{sign}{#1}}\)</span>
<span class="math">\(\newcommand\dist[1]{\F{distance}{#1}}\)</span>
<span class="math">\(\newcommand\DTF{{\Delta t}}\)</span>
<span class="math">\(\newcommand\IL[1]{\normalsize{#1}}\)</span>
<span class="math">\(\newcommand\MX[2]{\F{max}{#1,#2}}\)</span>
<span class="math">\(\newcommand\adr[1]{\F{adaptive}{#1}}\)</span>
<span class="math">\(\newcommand\clip[3]{\F{clip}{#1,#2,#3}}\)</span>
<span class="math">\(\newcommand\supply[1]{\F{supply}{#1}}\)</span>
<span class="math">\(\newcommand\iss[1]{\F{issuance}{#1}}\)</span>
<span class="math">\(\newcommand\isb[1]{\F{issuance_{block}}{#1}}\)</span>
<span class="math">\(\newcommand\tw{\Sigma_w}\)</span>
<span class="math">\(\newcommand\rw[2]{\F{reward_{#1}}{#2}}\)</span>
<span class="math">\(\newcommand\tip[2]{\F{tip_{#1}}{#2}}\)</span>
<span class="math">\(\newcommand\lbs[1]{\F{subsidy_{LB}}{#1}}\)</span>
<span class="math">\(\newcommand\exp[1]{\F{exp}{#1}}\)</span>
<span class="math">\(\newcommand{\vdf}{\mathit{VDF}}\)</span></p>
<p><em><span class="caps">TL</span>;<span class="caps">DR</span> This document provides a high-level functional specification
for <a href="#adaptive-issuance">Adaptive Issuance</a> and
<a href="#new-staking-mechanism">Staking</a>, two new features of the Oxford
protocol upgrade proposal, which together constitute a major evolution
of Tezos’ Proof-of-Stake mechanism.</em></p>
<ul>
<li><strong><span class="caps">EDITED</span> on 02/10/2023</strong> <em>to correct the value for <span class="math">\(\IL{\grf}\)</span><sup id="fnref2:grf"><a class="footnote-ref" href="#fn:grf">1</a></sup>
in the </em><em><a href="#dynamic_rate">dynamic rate</a></em><em> formula, which did not
reflect the value used in its implementation <a href="https://research-development.nomadic-labs.com/oxford-announcement.html">in
Oxford</a>.</em></li>
</ul>
<h1 id="adaptive-issuance"><a class="toclink" href="#adaptive-issuance">Adaptive Issuance</a></h1>
<p>Adaptive Issuance is a novel mechanism regulating tez issuance in Tezos.</p>
<p>Currently, the <a href="https://tezos.gitlab.io/active/protocol.html">Tezos economic
protocol</a> issues new tez via:</p>
<ul>
<li>Participation rewards: incentives given to delegates for
<a href="https://tezos.gitlab.io/active/consensus.html#rewards">participation in
consensus</a>
and random seed generation.</li>
<li>The <a href="https://tezos.gitlab.io/active/liquidity_baking.html">Liquidity Baking (<span class="caps">LB</span>)
subsidy</a>.</li>
<li>Protocol “invoices”: lump sums of tez issued and allocated during
protocol migration.</li>
</ul>
<p>Participation rewards and the <span class="caps">LB</span> subsidy are regularly issued by the
protocol, whereas the value and recipients of invoices are defined
discretionarily by the developers of a protocol proposal. In the
Nairobi protocol, and in previous ones, the values for participation
rewards and the <span class="caps">LB</span> subsidy, if any, are defined by the Tezos protocol
using fixed constants.</p>
<p>The <a href="https://research-development.nomadic-labs.com/oxford-announcement.html">Oxford protocol
proposal</a>
introduces Adaptive Issuance: a mechanism where the amount of
<em>regularly</em> issued tez (participation rewards and the <span class="caps">LB</span> subsidy, if
active) depends on the global <strong>staked funds ratio</strong> — that is, the
ratio of staked tez to the total supply. This lets issuance roughly
match the <em>actual</em> security budget the chain requires, the amount
needed to encourage participants to stake and produce blocks, but <em>no
more</em>.</p>
<p>At the end of each blockchain
<a href="https://tezos.gitlab.io/active/glossary.html#cycle">cycle</a>, the
regular issuance is adjusted, to nudge the staked funds ratio towards
a protocol-defined target (set at 50% in Oxford). Participation
rewards and the <span class="caps">LB</span> subsidy are recomputed to match that budget. When
the staked funds ratio decreases and diverges from the target,
emission rates increase, incentivizing participants to stake funds to
re-approach the target. Conversely, incentives decrease as the ratio
increases beyond the target.</p>
<h2 id="adaptive-issuance-rate"><a class="toclink" href="#adaptive-issuance-rate">Adaptive issuance rate</a></h2>
<p>The <strong><a href="#adaptive_rate">adaptive issuance rate</a></strong>
determines, at the end of cycle <span class="math">\(\IL{c}\)</span>, the issuance for cycle
<span class="math">\(\IL{c + 5}\)</span>. The adaptive issuance rate is the sum of a <strong><a href="#static_rate">static
rate</a></strong> and a <strong><a href="#dynamic_rate">dynamic
rate</a></strong>. The final result is clipped to
ensure nominal emissions remain within <span class="math">\(\IL{[\minR,\ \maxR]}\)</span> (set to
[0.05%, 5%] in Oxford) of the total supply.</p>
<figure align="center" id="Figure-1"> <img
src="/images/ai/adaptive_rate.png" alt="Figure 1 plots the nominal
issuance rate and the static rate as a function of the staked
ratio f. We pick the value 0.0075 (or 0.75%) for the dynamic rate,
but this is just an example and that number varies dynamically
over time." width="90%"> </a> <figcaption align="center">
<b>Figure 1:</b> adaptive issuance rate as a function of the
staked funds ratio f.</figcaption> </figure>
<p>Figure 1 plots the nominal issuance rate and the static rate as a
function of the staked ratio <span class="math">\(\IL{f}\)</span>. In the graph above, we picked
the value 0.0075 (or 0.75%) for the dynamic rate, but this is just an
example and that number varies dynamically over time.</p>
<p>The <strong>static rate</strong> is a static mechanism, which approximates <a href="https://en.wikipedia.org/wiki/Dutch_auction">a Dutch
auction</a> to compute a
nominal issuance rate as a function of the staked funds ratio for a
given cycle. Its value decreases as the staked funds ratio increases,
and <em>vice versa</em>.</p>
<p><br>
<a id="static_rate"><strong><span class="caps">STATIC</span> <span class="caps">RATE</span></strong></a> Let <span class="math">\(\IL{f}\)</span> be
the staked funds ratio at the end of cycle <span class="math">\(\IL{c}\)</span>. Then, the
<strong>static rate</strong> is defined as: </p>
<p><span class="math inline">
\(\static{f} = \sfr \tmult \frac{1}{f^2}\)
</span></p>
<p>The choice of <span class="math">\(\IL{\sfr}\)</span> as a scaling factor ensures that the curve
takes reasonable values for plausible staking ratios. Moreover,
assuming Adaptive Issuance is activated with a dynamic ratio of 0, and
at current staked funds ratio (that is, ~7.5% of the total supply),
this factor allows for a smooth transition from current issuance rate (~4.6%).</p>
<p>The <strong>dynamic reward rate</strong> adjusts itself over time based on the
distance between the staked funds ratio <span class="math">\(\IL{f}\)</span> and the 50% (±2%)
target ratio (<span class="math">\(\IL{\tc}\)</span> and <span class="math">\(\IL{\tr}\)</span> parameters below), increasing
when <span class="math">\(\IL{f}\)</span> < 48% and decreasing when <span class="math">\(\IL{f}\)</span> > 52%, provided the
total issuance rate is not hitting its lower or upper limit.</p>
<!-- \begin{align*} -->
<!-- <span class="amp">&</span> \dyn{c} =\ \dyn{c -1} + \sgn{\tc - \F{f}{c}} \tmult \grf \tmult \dist{\F{f}{c}} \tmult \DTF \\ -->
<!-- <span class="amp">&</span> \dyn{c_0} =\ 0 -->
<!-- \end{align*} -->
<p><br>
<a id="dynamic_rate"><strong><span class="caps">DYNAMIC</span> <span class="caps">RATE</span></strong></a> The <strong>dynamic rate</strong> <span class="math">\(\IL{\dyn{c}}\)</span> is defined at the end of cycle <span class="math">\(\IL{c}\)</span> as:<br></p>
<p>
<span class="math inline">
\(\dyn{c} = \dyn{c -1} + \sgn{\tc - \F{f}{c}} \tmult \grf \tmult \dist{\F{f}{c}} \tmult {\Delta t}\)
<br>
<span class="math inline">
\(\dyn{c_0} = 0\)
</span>
</p>
<p><span class="math">\(\IL{\dyn{c}}\)</span> is then clipped to <span class="math">\(\IL{\left[ 0, \maxR - \static{\F{f}{c}}\right]}\)</span>, ensuring that <span class="math">\(\IL{\static{\F{f}{c}} + \dyn{c} \leq \maxR}\)</span>.
<br></p>
<p>In this formula:</p>
<ul>
<li><span class="math">\(\IL{c_0}\)</span> is the first cycle where Adaptive Issuance is active.</li>
<li>
<p>Given a cycle <span class="math">\(\IL{c}\)</span>, <span class="math">\(\IL{\F{f}{c}}\)</span> denotes the <strong>staked funds
ratio</strong> at the end of the cycle, and <span class="math">\(\IL{\dyn{c}}\)</span> the value of the
dynamic rate computed in that cycle.</p>
</li>
<li>
<p><span class="math">\(\IL{\tc}\)</span> = 0.5 and <span class="math">\(\IL{\tr}\)</span> = 0.02 denote, respectively, the
<strong>target staked funds ratio</strong> and the <strong>radius</strong> of the interval
centered on the target ratio.</p>
</li>
<li>
<p><span class="math">\(\IL{\grf}\)</span> = 0.01, controls the speed at which the dynamic rate
adjusts. The value is set so that a one percentage point deviation
of the staked funds ratio changes the dynamic rate by 0.01
percentage points per day<sup id="fnref:grf"><a class="footnote-ref" href="#fn:grf">1</a></sup>.</p>
</li>
<li>
<p><span class="math">\(\IL{\dist{\F{f}{c}} = \MX{0}{\left|\F{f}{c} - \tc \right| - \tr}}\)</span>
denotes the (<em>absolute</em>) distance between the staked funds ratio
<span class="math">\(\IL{\F{f}{c}}\)</span> and the interval <span class="math">\(\IL{\left[ \tc - \tr, \tc + \tr
\right]}\)</span>.</p>
</li>
<li>
<p><span class="math">\(\IL{\DTF = \frac{16384 \tmult 15}{86400} = 2.8\overline{444}}\)</span>,
denotes the minimal duration (in days) of a Tezos cycle, assuming
all 16384 blocks in the cycle are produced at the minimal allowed
time – that is, every 15 seconds.</p>
</li>
<li>
<p><span class="math">\(\IL{\sgn{\tc - \F{f}{c}} = 1}\)</span> if <span class="math">\(\IL{\F{f}{c} \leq \tc}\)</span> and
<span class="math">\(-1\)</span> otherwise, denotes the sign of the distance between the target
ratio <span class="math">\(\IL{\tc}\)</span> and the staked funds ratio <span class="math">\(\IL{\F{f}{c}}\)</span>.</p>
</li>
</ul>
<p>In a nutshell, <span class="math">\(\IL{\dyn{c}}\)</span> increases and decreases by an amount
proportional to the distance between the target rate and the interval
<span class="math">\(\IL{\left[ \tc - \tr, \tc + \tr \right]}\)</span>, while ensuring that the
adaptive issuance rate is kept within <span class="math">\(\IL{[\minR,\ \maxR]}\)</span> bounds.</p>
<p>Finally, as mentioned before, the nominal adaptive issuance rate<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">2</a></sup>
for a cycle <span class="math">\(\IL{c + 5}\)</span> is defined as the sum of the static rate and
the dynamic rate, clipped to stay within 0.05% — 5% range.</p>
<p><br>
<a id="adaptive_rate"><strong><span class="caps">ADAPTIVE</span> <span class="caps">ISSUANCE</span> <span class="caps">RATE</span></strong></a> Let <span class="math">\(\F{f}{c}\)</span> be the staked funds ratio at the end of cycle <span class="math">\(\IL{c}\)</span>, the <strong>adaptive issuance rate</strong> for cycle <span class="math">\(\IL{c+5}\)</span> is defined as: </p>
<p>
<span class="math inline">
\(\adr{c + 5} = \clip{\dyn{c} + \static{\F{f}{c}}}{\minR}{\maxR}\)
</p>
<p><br></p>
<h2 id="adaptive-rewards"><a class="toclink" href="#adaptive-rewards">Adaptive rewards</a></h2>
<p>In Nairobi, and in previous Tezos protocols, participation rewards and
the <span class="caps">LB</span> subsidy are fixed values defined by protocol constants. With
the proposed mechanism, the <a href="#adaptive_rate">adaptive issuance rate</a>
provides instead a budget for the whole cycle, which gets allocated
equally to each block of the cycle and distributed between the various
rewards, in proportion to their relative
<a href="#reward_weights">weights</a>.</p>
<p><br>
<a id="block_issuance"><strong><span class="caps">ADAPTIVE</span> <span class="caps">ISSUANCE</span> <span class="caps">PER</span> <span class="caps">BLOCK</span></strong></a> Let <span class="math">\(\supply{c}\)</span> be the total supply at the end of cycle <span class="math">\(\IL{c}\)</span>, the <strong>issuance per block</strong> for cycle <span class="math">\(\IL{c+5}\)</span> is defined as: </p>
<p>
<span class="math inline">
\(\isb{c + 5} = \frac{\adr{c + 5}}{2102400} \tmult \supply{c}\)
</p>
<p>Where 2102400 = <span class="math">\(\IL{\frac{365 \tmult 24 \tmult 60 \tmult 60}{15}}\)</span> is
the maximal number of blocks produced in a year, given a minimal block
time of 15 seconds.</p>
<p><br>
<a id="reward_weights"><strong><span class="caps">REWARD</span> <span class="caps">WEIGHTS</span></strong></a> The Oxford proposal defines the weights for participation rewards and the <span class="caps">LB</span> subsidy as:</p>
<ul>
<li>Attestation (formerly, endorsing) rewards : 10,240.</li>
<li>Fixed baking reward: 5,120.</li>
<li>Bonus baking reward: 5,120.</li>
<li><span class="caps">LB</span> subsidy: 1,280.</li>
<li>Nonce revelation tip: 1.</li>
<li><span class="caps">VDF</span> tip: 1.</li>
</ul>
<p>The total sum of all weights is <span class="math">\(\tw\)</span> = 21762. The total issuance per
block, <span class="math">\(\IL{\isb{c}}\)</span>, is distributed amongst the different rewards in
proportion to their weight.</p>
<p><strong>Consensus rewards.</strong> Since the adoption of Tenderbake, Tezos
protocols have rewarded delegates <a href="https://tezos.gitlab.io/oxford/consensus.html#rewards">for their participation in
consensus</a> with
the following rewards per block:</p>
<ul>
<li>A fixed <strong>baking</strong> reward, given to the delegate which produced the
<em>payload</em> of the block (i.e. choosing transactions, and other
non-consensus operations).</li>
<li>A variable, baking <strong>bonus</strong> reward given to the delegate which
produced the block included in the chain. This bonus is given for
including attestations, if their combined attesting power exceeds
the minimal threshold (two thirds of total slots).</li>
<li>A <em>collective</em> <strong>attestation</strong> reward shared between the delegates
selected in the consensus committee for that block level, for
attesting block proposals.</li>
</ul>
<p>We refer to <a href="https://tezos.gitlab.io/oxford/consensus.html">the technical
documentation</a> for
further insight on the pre-requisites and distribution of these
rewards. Here, we derive the new formulas which compute their values
<em>per block</em> for a cycle <span class="math">\(\IL{c}\)</span>:</p>
<p>
<span class="math inline">
\(\rw{baking}{c} = \rw{bonus}{c} = \frac{5120}{\tw} \tmult \isb{c}\)
<br>
<span class="math inline">
\(\rw{attestation}{c} = \frac{10240}{\tw} \tmult \isb{c}\)
</span>
</p>
<p>Note that these formulas change the value of available rewards, but
not why and how they are awarded. Hence, <span class="math">\(\IL{\rw{bonus}{c}}\)</span> still
denotes the maximal value for this reward: the actual reward issued
depends on the total number of attested slots in a block. Similarly,
<span class="math">\(\IL{\rw{attestation}{c}}\)</span> is also a maximal value per block, which is
further shared between multiple delegates depending on the number of
attested slots and subject to the existing participation conditions.</p>
<p><strong>Nonce and <span class="caps">VDF</span> revelation tips.</strong> The rewards allocated to delegates
for contributing to <a href="https://tezos.gitlab.io/oxford/randomness_generation.html#randomness-generation">random seed
generation</a>
(that is for, revealing nonce seeds and posting <span class="caps">VDF</span> proofs) are not
paid each block, but rather every 128 blocks. The adjusted formulas result:</p>
<p>
<span class="math inline">
\( \tip{\vdf}{c} = \tip{nr}{c} = 128 \tmult \frac{1}{\tw} \tmult \isb{c}\)
</span>
</p>
<p><strong>Liquidity baking subsidy.</strong> The <a href="https://tezos.gitlab.io/oxford/liquidity_baking.html"><span class="caps">LB</span>
subsidy</a> per
block is determined by the following formula:</p>
<p>
<span class="math inline">
\(\lbs{c} = \frac{1280}{\tw} \tmult \isb{c}\)
</span>
</p>
<p>Note that while the subsidy is issued <strong>only if</strong> the feature is on,
its weight is always counted in the computation of <span class="math">\(\IL{\tw}\)</span>. In
other words, the budget for the <span class="caps">LB</span> subsidy is always allocated,
regardless of whether it is issued or not.</p>
<p>The Oxford protocol proposal implements a new <a href="https://tezos.gitlab.io/oxford/rpc.html#get-block-id-context-issuance-expected-issuance"><span class="caps">RPC</span>
endpoint</a>,
<code>/issuance/expected_issuance</code>, which reports the precomputed values
of all participation rewards and the <span class="caps">LB</span> subsidy, for the cycle
corresponding to the queried block level, and the next 4 cycles.</p>
<h1 id="new-staking-mechanism"><a class="toclink" href="#new-staking-mechanism">New Staking mechanism</a></h1>
<p>Staking is an evolution of the existing Tezos <a href="https://tezos.gitlab.io/oxford/proof_of_stake.html">Liquid Proof-of-Stake
mechanism</a>. It
introduces a new role for network participants, called <strong>staker</strong>,
complementary to the existing
<a href="https://tezos.gitlab.io/oxford/glossary.html#delegate"><em>delegate</em></a>
(also known as <em>baker</em>) and <em>delegator</em> roles. A staker must also be a
<em>delegator</em> — that is, they must first choose a delegate.</p>
<p>When stakers <strong>stake </strong>funds towards a delegate’s <strong>staking</strong>
<strong>balance</strong>, the associated <strong>baking</strong> and <strong>voting powers</strong> accrue to
that delegate. Similarly to how delegated funds work, staked funds
remain within the staker’s account at all times.</p>
<p>Staked and delegated funds <strong>have different weights</strong> in the
computation of delegates’ baking and voting powers: staked funds (both
external stakes by stakers and the delegate’s own) count <strong>twice</strong> as
much as delegated funds.</p>
<p>Unlike delegated funds, staked funds are considered to contribute to
the security deposit associated with their chosen delegate. Thus, they
are subject to
<a href="https://tezos.gitlab.io/oxford/consensus.html#slashing">slashing</a> if
the delegate misbehaves by
<a href="https://tezos.gitlab.io/oxford/glossary.html#Doublesigning">double-signing</a>
block proposals or consensus operations, and are subject to the same
withdrawal delays — colloquially, they are “frozen”.</p>
<p>Stakers are slashed proportionally to their contribution to the
delegate’s staking balance. To simplify slashing, double-baking
penalties are now proportional to staked funds: instead of the
previous fixed sum of 640 tez they are now set to 10% of the
delegate’s stake. Moreover, denunciation rewards (both for
double-baking and double-attestations) are reduced from one half to
one seventh of the slashed funds. The chosen value prevents
adversarial delegates from abusing the slashing mechanism for profit
at the expense of their stakers.</p>
<p><em>Delegates</em> <a href="#staking-policy-configuration">configure their staking
policy</a> by setting staking parameters
which regulate whether they accept stakers (the default being to
reject them), and if so, up to which fraction of their total staking
balance. They can also configure which proportion of the staking
rewards is set to accrue to their own staked balance versus their
unfrozen, spendable balance. As <a href="#adaptive-rewards">participation
rewards</a> are paid to the staked balance, and
automatically shared between delegates and their stakers, delegates
can use this parameter to collect an <em>edge</em> from the rewards
attributable to their stakers.</p>
<p>If and when Oxford activates, freezing and unfreezing of staked funds
will be controlled directly by delegates and stakers, and will no
longer be automatic. This entails that staked funds are frozen until
manually unfrozen by stakers. This is a two step process which spans
for at least 7 cycles (cf. <a href="#staked-funds-management">Staked funds
management</a>).</p>
<p>A new user interface is provided for delegates and stakers to interact
with the mechanism. It is based on four <em>pseudo-operations</em>: <code>stake</code>,
<code>unstake</code>, <code>finalize_unstake</code>, and <code>set_delegate_parameters</code>.
Pseudo-operations are self-transfers: a transfer operation where the
destination matches the source – each involving a special entry-point
of the same name introduced for <a href="https://tezos.gitlab.io/oxford/glossary.html#implicit-account">implicit
accounts</a>. This
approach was chosen to minimize the work required by wallets,
custodians, exchanges, and other parties to support the functionality.</p>
<p><strong><span class="caps">NB</span> </strong> <a href="#feature-activation-vs-protocol-activation">Until feature
activation</a>: only
<em>delegates</em> can stake funds and the relative weight of staked and
delegated funds remains unchanged. In the current implementation, only
<em>implicit accounts</em> can become stakers. In other words, smart
contracts cannot stake funds (they can of course still delegate them).</p>
<h2 id="staking-policy-configuration"><a class="toclink" href="#staking-policy-configuration">Staking policy configuration</a></h2>
<p><em>Delegates</em> can configure their staking policy by setting the
following parameters:</p>
<ul>
<li><code>edge_of_baking_over_staking</code>: a ratio between 0 and 1, whose
default value is 1. This parameter determines the fraction of the
rewards that accrue to the delegate’s liquid spendable balance —
the remainder accrues to frozen stakes.</li>
<li><code>limit_of_staking_over_baking</code>: a non-negative number, denoting the
maximum portion of external stake by stakers over the delegate’s own
staked funds. It defaults to 0 — which entails that delegates do
not accept external stakes by default. It is moreover capped by a
global constant, set to 5 in Oxford, which ensures the baker
controls a significant part of the stake.</li>
</ul>
<p>Delegates can modify these staking parameters at all times, using the
<code>set_delegate_parameters</code> pseudo-operation: that is, by transferring 0
tez to their own <code>set_delegate_parameters</code>. The chosen values for both
parameters need to be supplied. The new parameters are then applied 5
cycles later.</p>
<p><strong>On overstaking and overdelegation.</strong> Note that if a delegate’s
<code>limit_of_staking_over_baking</code> is exceeded (that is, the delegate is
<em>overstaked</em>), the exceeding stake is automatically considered a
<em>delegation</em> for the delegate’s baking and voting power calculation,
but it does remain slashable. The new mechanism does not alter
<em>overdelegation</em> (delegated funds beyond 9 times the delegate’s own
stake) nor its consequence on voting and baking powers. That is,
overdelegated funds are not counted towards a delegate baking power,
but they do increase their voting power.</p>
<h2 id="staked-funds-management"><a class="toclink" href="#staked-funds-management">Staked funds management</a></h2>
<p>Stakers (and delegates) can use the <code>stake</code>, <code>unstake</code>, and
<code>finalize_unstake</code> pseudo-operations to control their stakes. <a href="#Figure-2">Figure
2</a> illustrates their effect on a staker’s funds. Note that
while these pseudo-operations change the <em>state</em> of the involved
funds, they remain otherwise within the staker’s account at all times.</p>
<figure align="center" id="Figure-2"> <img
src="/images/ai/staked_funds_transitions.png" alt="Staked funds
management using pseudo-operations." width="90%"> </a> <figcaption
align="center"> <b>Figure 2:</b> staked funds management using
pseudo-operations.</figcaption> </figure>
<p>To <em>stake</em> funds, a delegator uses the <code>stake</code> pseudo-operation,
transferring the chosen amount of <strong>spendable</strong> tez to their own
<code>stake</code> entry-point. The <strong>staked</strong> tez will then be frozen and
contribute to their chosen delegate’s staking balance. Note that the
<code>stake</code> pseudo-operation will fail if the sender account is not
<em>delegated</em>.</p>
<p>To <em>unstake</em> funds, a staker first submits an unstake request with the
<code>unstake</code> pseudo-operation. This is implemented by transferring 0 tez
to their <code>unstake</code> entrypoint, while passing the chosen amount as a parameter.</p>
<p>The requested amount will be <strong>unstaked</strong> but will remain <strong>frozen</strong>.
After 7 cycles, unstaked frozen tokens are no longer considered at
stake nor slashable. They are said then to be both <strong>unstaked</strong> and
<strong>finalizable</strong>.</p>
<p>A staker can retrieve all unstaked and finalizable tokens at any time,
making them spendable again. This is done using the <code>finalize_unstake</code>
entrypoint — that is, by transferring 0 tez to their
<code>finalize_unstake</code> entry-point.</p>
<h1 id="feature-activation-vs-protocol-activation"><a class="toclink" href="#feature-activation-vs-protocol-activation">Feature activation vs protocol activation</a></h1>
<p>Should the Oxford protocol proposal be accepted by the community, and
once the protocol becomes active on Tezos Mainnet, most of the
features described in this document will <strong>not</strong> be enabled by
default, only latent possibilities in the protocol, waiting for a
separate activation.</p>
<p>In particular, the following changes will require additional approval
from delegates via separate feature activation vote mechanism:</p>
<ul>
<li>Adaptive issuance — including notably the changes to the
computation of consensus rewards and the <span class="caps">LB</span> subsidy.</li>
<li>Ability for <em>delegators</em> to become <em>stakers</em> — until feature
activation delegates continue to be the only participants who can
<strong>stake</strong> funds.</li>
<li>The changes in weight for staked and delegated funds towards the
computation of baking and voting rights.</li>
</ul>
<p>Other changes described earlier would be enabled from Oxford’s activation:</p>
<ul>
<li>The new interface for stake manipulation based on
<em>pseudo-operations</em>. Note that this entails the deprecation of the
<code>set/unset deposits limit</code> interface and also the end of automatic
deposit freezing. On protocol activation, each delegate’s stake is
derived from the frozen deposits at the end of the last cycle of Nairobi.</li>
<li>The changes in slashing penalties (double-baking penalties are set
to 10% of the staked funds) and denunciation rewards (they amount to
one seventh of slashed funds).</li>
<li>Changes to protocol constants. Note that this entails calculating
participation rewards and the <span class="caps">LB</span> subsidy using the weight-based
formulas, but these are defined so that they match the previous
values when <a href="#adaptive-issuance">Adaptive Issuance</a> is not active.</li>
</ul>
<h2 id="activation-vote"><a class="toclink" href="#activation-vote">Activation Vote</a></h2>
<p>We highlight the following principles behind the feature activation
vote mechanism:</p>
<ul>
<li>If and when Oxford activates, delegates can start voting for
(<strong>On</strong>) or against (<strong>Off</strong>) the feature activation of the changes
listed above in each block they bake. They can also abstain with a
<strong>Pass</strong> vote.</li>
<li>These votes are cast by block-producing delegates, and are included
in block headers.</li>
<li>Participation is not mandatory, defaulting to <strong>Pass</strong> in the
absence of signaling.</li>
<li>The feature activation vote has two phases: a <em>Voting</em> phase and a
subsequent <em>Adoption</em> phase.</li>
<li>The <em>Voting</em> phase is driven by an Exponential moving average (<span class="caps">EMA</span>)
whose <em>half-life</em> is 2 weeks. That is, it takes two weeks for the
<span class="caps">EMA</span> to raise from 0% to 50% assuming only <strong>On </strong>votes are cast.</li>
<li>The target threshold is a supermajority of 80% of <strong>On</strong> votes over
<strong>On plus Off</strong> votes.</li>
<li>There is no time limit or fixed duration for the Voting phase. It
continues as long as the threshold is not met. There is no <em>quorum</em>
either, the lack of participation (reified as <strong>Pass</strong> votes) is not
taken into account by the <span class="caps">EMA</span>, and hence only affects the time
required to reach the threshold.</li>
<li>If the threshold is met, the Voting phase will complete at the end
of the current cycle, and the Adoption phase will start at the
beginning of the following cycle.</li>
<li>The Adoption phase lasts 7 cycles. The beginning of the cycle
following the end of the Adoption phase activates the guarded features.</li>
<li>There is <strong>no automatic deactivation</strong> of the guarded features once
in (and after) the Adoption phase — subsequent votes continue to be
counted towards an updated <span class="caps">EMA</span>, but without any further effect.</li>
</ul>
<p><strong><span class="caps">NB</span></strong> In the implementation in the Oxford protocol, the issuance rate
is computed 5 cycles in advance. Thus, in the first 5 cycles where is
active, the protocol does not use the <a href="#adaptive-rewards">adaptive reward
formula</a> and keeps using the current reward values.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:grf">
<p><span class="caps">EDITED</span> on 02/10/2023 to correct the value for <span class="math">\(\IL{\grf}\)</span>. The
original version mistakenly defined it as 0.0001. <a class="footnote-backref" href="#fnref:grf" title="Jump back to footnote 1 in the text">↩</a><a class="footnote-backref" href="#fnref2:grf" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:1">
<p>Note that if the nominal annual issuance rate is <span class="math">\(r\)</span>, the
annualized rate is close to <span class="math">\(\IL{\exp{r} - 1}\)</span> as it is
compounded at every cycle. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>EVM Rollups Are Coming to Tezos – Now on Testnet2023-07-18T18:00:00+02:002023-07-18T18:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2023-07-18:/evm-tezos-testnet.html<p>Run Solidity smart contracts on a Tezos rollup.</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: An <span class="caps">EVM</span>-compatible Smart Rollup is available on Ghostnet,
Tezos’ permanent testnet. This blog post shows you how to get started
with it.</strong></p>
<p>Connect your Metamask wallet, deploy Solidity contracts with Remix,
build dApps using Ethereum <span class="caps">JSON</span>-RPCs, and still benefit from the
latest Tezos innovations!</p>
<p>We are happy to present <span class="caps">EVM</span>-compatibility on Tezos via an <a href="https://gitlab.com/tezos/tezos/-/tree/master/src/kernel_evm">open source
<span class="caps">EVM</span> Smart
Rollup</a>
developed by Trilitech, Marigold, Functori and Nomadic Labs. As the
next step towards Mainnet-readiness, an instance of this rollup has
been deployed on the <a href="https://teztnets.xyz/ghostnet-about">Ghostnet test
network</a> at block level
<a href="https://ghostnet.tzkt.io/oo8KAmTNYNExKGHSZ2Da5Nk3jCgw2dNam8TVxQsAaeCName1pzi/17509065">3289331</a>.</p>
<p>This deployment is a <em>community</em> rollup, meaning it is an
<span class="caps">EVM</span>-compatible execution environment where all interested community
members and project developers can explore and experiment with
<span class="caps">EVM</span>-development on Tezos.</p>
<p>With that goal in mind, this blog post provides instructions on how to
interact with the deployed rollup using the <a href="https://tezos.gitlab.io/introduction/howtouse.html">Octez software
suite</a>.</p>
<h2 id="whats-an-evm-compatible-smart-rollup"><a class="toclink" href="#whats-an-evm-compatible-smart-rollup">What’s an <span class="caps">EVM</span>-compatible Smart Rollup?</a></h2>
<p>The Ethereum<a href="https://ethereum.org/en/developers/docs/evm/"> Virtual Machine
(<span class="caps">EVM</span>)</a> is the execution
environment in which all Ethereum accounts and smart contracts
live. An <span class="caps">EVM</span>-compatible <a href="https://tezos.gitlab.io/alpha/smart_rollups.html">Smart
Rollup</a> is a rollup
whose <em>kernel</em> implements an <span class="caps">EVM</span>-compatible execution
environment. This enables Ethereum smart contracts to be
frictionlessly deployed and executed on Tezos.</p>
<p>Because Smart Rollups allow developers to write applications written
in any programming language that compiles to <span class="caps">WASM</span>, we were able to
develop an open source kernel using an established implementation of
the <span class="caps">EVM</span> in Rust,
<a href="https://github.com/rust-blockchain/evm">SputnikVM</a>. This kernel
creates a separate blockchain running in Tezos’ Layer 2, producing its
own blocks and processing its own transactions.</p>
<p>Currently, there is one <span class="caps">EVM</span> block per Tezos block, but Smart Rollups
are upgradable, and nothing prevents us from having different block
times from Layer 1 in the future. We retrieve the list of transactions
to include in <span class="caps">EVM</span>-rollup blocks by reading the Smart Rollups’ <a href="https://tezos.gitlab.io/alpha/smart_rollups.html#rollups-inbox"><em>shared
inbox</em></a>. Thus,
we use the Tezos network as our consensus layer, as it provides a
sequence of transactions to include in Layer 2 blocks.</p>
<p>The <span class="caps">EVM</span> kernel is executed by an Octez Smart Rollup node, which is
kernel agnostic and not equipped to communicate with Ethereum wallets
or block explorers out of the box. For this, we provide another layer
on top of the Octez rollup node in the form of an <span class="caps">EVM</span> node.</p>
<p>This facade node is provided as a new Octez binary. It currently
partially supports <a href="https://ethereum.org/en/developers/docs/apis/json-rpc/">Ethereum’s <span class="caps">JSON</span>-<span class="caps">RPC</span>
<span class="caps">API</span></a>
specification, but we aim to provide full support in the future, so as
to achieve full compatibility with the Ethereum tooling ecosystem.</p>
<h2 id="getting-started-with-the-evm-rollup"><a class="toclink" href="#getting-started-with-the-evm-rollup">Getting started with the <span class="caps">EVM</span> rollup</a></h2>
<p>For reasons well-explained <a href="https://ctez.app/faq">here</a>, we have
decided to use Ctez as the native token for this deployment of the <span class="caps">EVM</span>
rollup. Therefore, your balance will be given in ctez, you’ll transfer
ctez, and you’ll pay fees with ctez.</p>
<p><strong>1. Get Ctez tokens from the faucet</strong></p>
<p>The first step is getting ahold of sufficient Ctez tokens on Layer 1,
which in this case is Ghostnet. In this tutorial, we use the Tezos
address <code>tz1g2RbQtxZRHw8oj4oSnhRzy7iRA2hxU4TD</code> and the Ethereum
address <code>0x8aaD6553Cf769Aa7b89174bE824ED0e53768ed70</code>. All you have to
do is replace these with your respective accounts.</p>
<p>For the deployment of the rollup on Ghostnet, we picked <a href="https://better-call.dev/ghostnet/KT1Q4ecagDAmqiY3ajvtwfNZyChWy86W7pzb/operations">this instance
of the Ctez
contract</a>. To
speed up user onboarding, Marigold has integrated support for Ctez in
their <a href="https://faucet.marigold.dev/">test network’s faucet</a>.</p>
<p><strong>2. Authorize the Layer 1 bridge to deposit your Ctez tokens</strong></p>
<p>The <span class="caps">EVM</span> rollup will accept deposits only from a <a href="https://ghostnet.tzkt.io/KT1HJphVV3LUxqZnc7YSH6Zdfd3up1DjLqZv/operations/">specific Layer 1
contract</a>,
responsible for bridging Ctez tokens to the Layer 2 native tokens,
called <em>the Layer 1 bridge</em>.</p>
<p>The Layer 1 bridge needs to be able to transfer Ctez tokens from your
account to itself. As the contract is compliant with the <a href="https://tezos.gitlab.io/user/fa12.html#managing-tokens"><span class="caps">FA1</span>.2
specification</a>,
we do this by creating a token allowance for the bridge contract,
using the <code>%approve</code> entry point:</p>
<p><code>./octez-client --endpoint https://rpc.ghostnet.teztnets.xyz from fa1.2 contract KT1Q4ecagDAmqiY3ajvtwfNZyChWy86W7pzb as tz1g2RbQtxZRHw8oj4oSnhRzy7iRA2hxU4TD approve <amount-to-deposit> from KT1HJphVV3LUxqZnc7YSH6Zdfd3up1DjLqZv --burn-cap 0.0175</code></p>
<p>This call will enable the bridge contract to take (up to)
<code><amount-to-deposit></code> ctez from your account and transfer them to
itself, in the next step. Note that the amount should be given in
µctez (10<sup>-6</sup> ctez), e.g. if you want to deposit 100 ctez,
you need to put in 100,000,000 µctez.</p>
<p><strong>3. Deposit your Ctez tokens on Layer 2 via the Layer 1 bridge</strong></p>
<p>The Layer 1 bridge exposes an entrypoint <code>%deposit</code>, enabling users to
deposit ctez. The entrypoint takes 3 parameters:</p>
<ul>
<li>The receiver’s <span class="caps">EVM</span> address. Note that the user must make sure this
is a <a href="https://support.metamask.io/hc/en-us/articles/4702972178459-The-Ethereum-address-format-and-why-it-matters-when-using-MetaMask">valid Ethereum
address</a>.</li>
<li>The amount of ctez to deposit.</li>
<li>The maximum fee per unit of gas the sender is willing to pay, in Wei.</li>
</ul>
<p>You can discover the current gas price by using the <span class="caps">RPC</span>
<a href="https://ethereum.org/en/developers/docs/apis/json-rpc/#eth_gasprice">eth_gasPrice</a>
(note that the result is hex-encoded), e.g.</p>
<p><code>curl -X POST -H 'Content-Type: application/json' --data '{"jsonrpc":"2.0","id":35,"method":"eth_gasPrice","params":[]}' https://evm.ghostnet-evm.tzalpha.net/</code></p>
<p>After choosing values for the 3 parameters above, you can call the
entrypoint <code>%deposit</code>. In this example we’ve set the maximum fee per
unit of gas to 21000 Wei:</p>
<p><code>./octez-client --endpoint https://rpc.ghostnet.teztnets.xyz transfer 0 from tz1g2RbQtxZRHw8oj4oSnhRzy7iRA2hxU4TD to KT1HJphVV3LUxqZnc7YSH6Zdfd3up1DjLqZv --entrypoint "deposit" --arg "Pair (Pair <amount-to-deposit> 0x8aaD6553Cf769Aa7b89174bE824ED0e53768ed70) 21000" --burn-cap 0.1</code></p>
<h3 id="using-the-evm-rollup"><a class="toclink" href="#using-the-evm-rollup">Using the <span class="caps">EVM</span> rollup</a></h3>
<p>We have deployed a Ghostnet instance of
<a href="https://www.blockscout.com/">Blockscout</a>, a public Ethereum block
explorer, which is available at
<a href="https://explorer.ghostnet-evm.tzalpha.net/">https://explorer.ghostnet-evm.tzalpha.net/</a>. Once
you have deposited ctez to your <span class="caps">EVM</span> address, you can verify its
balance via the explorer, for example:
<a href="https://explorer.ghostnet-evm.tzalpha.net/address/0x8aaD6553Cf769Aa7b89174bE824ED0e53768ed70">https://explorer.ghostnet-evm.tzalpha.net/address/0x8aaD6553Cf769Aa7b89174bE824ED0e53768ed70</a>.</p>
<p>You are now ready to interact with the <span class="caps">EVM</span> rollup, whose public
<span class="caps">JSON</span>-<span class="caps">RPC</span> endpoint lies at
https://evm.ghostnet-evm.tzalpha.net. MetaMask is one of the most
popular wallets of the Ethereum ecosystem, and below is a quick guide
to using it with the <span class="caps">EVM</span> rollup deployed on Ghostnet. However, any
<span class="caps">EVM</span>-compatible wallet should work.</p>
<figure align="center" id="Figure-1"> <img
src="/images/evm-metamask.gif" alt= "Animation showing how to connect
Metamask with the EVM-rollup on Tezos Ghostnet." height="70%"
width="90%" align="center"> </figure>
<p>When you have connected your favorite wallet to the <span class="caps">EVM</span> rollup and
deposited funds, you can start to sign and send transactions via the
wallets. You can perform regular transfers between
<a href="https://ethereum.org/se/developers/docs/accounts/#types-of-account">externally-owned</a>
(non-contract) accounts:</p>
<figure align="center" id="Figure-2"> <img
src="/images/evm-transfer.gif" alt= "Animation demonstrating a
transfer of ctez on the Ghostnet EVM-rollup using Metamask."
height="70%" width="90%"> </figure>
<p>You can also deploy your own contracts and interact with them. Here is
an <a href="https://explorer.ghostnet-evm.tzalpha.net/address/0x705939Fd4C3bEEfc39a9C4517549BF33Bd0306d8">example of a
contract</a>
deployed on the <span class="caps">EVM</span> rollup.</p>
<h2 id="the-evm-rollup-is-work-in-progress"><a class="toclink" href="#the-evm-rollup-is-work-in-progress">The <span class="caps">EVM</span> rollup is work-in-progress</a></h2>
<p>The <span class="caps">EVM</span> rollup is still in <em>alpha</em>, and we are actively working on
making it feature complete and bug-free. You can follow the
development process directly on the <a href="https://gitlab.com/tezos/tezos">Tezos GitLab
repository</a>.</p>
<p>It is subject to frequent upgrades as new features are introduced and
existing ones are polished and hardened. However, these updates should
not affect end-user activity in the deployed <span class="caps">EVM</span>-compatible rollup.</p>
<p>Similarly to Ghostnet’s governance, a dictator key is implemented in
the <span class="caps">EVM</span> rollup and will be used to migrate the running instance, as
future and ongoing developments mature. When we reach the first stable
version of the <span class="caps">EVM</span> rollup, the dictator key will be removed and
replaced by an upgrade mechanism owned by Tezos Layer 1 governance.</p>
<p>If you feel more adventurous and want to try out the latest features
of the <span class="caps">EVM</span> rollup while they are being developed, it is possible to
experiment with the <span class="caps">EVM</span>-compatible rollup automatically deployed on
the <em>bleeding-edge</em> <a href="https://teztnets.xyz/dailynet-about">Dailynet test
network</a>. Check out the section
‘<span class="caps">EVM</span> Rollup’, and connect your tools with the endpoint:
<a href="https://evm.dailynet-YYYY-MM-DD.teztnets.xyz">https://evm.dailynet-<span class="caps">YYYY</span>-<span class="caps">MM</span>-<span class="caps">DD</span>.teztnets.xyz</a>.</p>
<h2 id="join-us-in-creating-the-best-possible-evm-experience"><a class="toclink" href="#join-us-in-creating-the-best-possible-evm-experience">Join us in creating the best possible <span class="caps">EVM</span> experience</a></h2>
<p>To advance the <span class="caps">EVM</span> from the current alpha state to a
production-ready solution, further testing is needed. We invite you to
experiment with the Ghostnet deployment and let us know your experience.</p>
<p>We would also like to hear from Ethereum/Solidity-focused developers
interested in deploying their project on an <span class="caps">EVM</span> rollup on Tezos. Reach
out to us on the <a href="https://tezos-dev.slack.com/join/shared_invite/zt-1yx8ipded-xSi2nxdgGsO6UXMa2FZS2Q#/shared-invite/email">Tezos-Dev Slack
workspace (in the channel #evm-testnets)</a>
and let’s talk about how we can help each other.</p>
<p>Finally, if you’re curious about the rollup-centric future of Tezos,
join us at the <a href="https://tezos.com/tez-dev/">TezDev</a> developer
conference in Paris on July 21 (right after EthCC). This year’s theme
is <a href="https://twitter.com/hashtag/RiseofRollups">#RiseofRollups</a>. Learn
about the future of Tezos, engage with relevant engineers about the
<span class="caps">EVM</span> roadmap – and witness a live demonstration of 1 million tps!</p>The Rollup Booster: A Data-Availability Layer for Tezos2023-07-06T18:00:00+02:002023-07-06T18:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2023-07-06:/data-availability-layer-tezos.html<p>A protocol-integrated, decentralized scaling stack for Tezos.</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: To achieve millions of <span class="caps">TPS</span>, Smart Rollups currently rely on
publishing data outside of the Tezos protocol. This blog post
introduces the next step: a protocol-level solution for rollup data
that is both highly scalable and fully decentralized.</strong></p>
<p>With the activation of Smart Rollups in the <a href="https://research-development.nomadic-labs.com/mumbai-preview.html#smart-rollups-activated">Mumbai protocol
upgrade</a>,
the Tezos ecosystem took a major step towards massive scalability. We
are thrilled to see the ecosystem already building and deploying Smart
Rollups on testnets, and we look forward to seeing them in action on Mainnet.</p>
<p>Meanwhile, at Nomadic Labs, Marigold, Functori and Trilitech, protocol
developers are busy building a next-level data solution for Smart Rollups.</p>
<p>The challenge is to have the Tezos protocol guarantee availability of
rollup transaction data without storing it in Layer 1 blocks. In this
blog post we explain our approach, starting with an overview of the
challenge at hand and then moving into a more technical explanation.</p>
<h2 id="whats-the-problem"><a class="toclink" href="#whats-the-problem">What’s the problem?</a></h2>
<p>The main feature of rollups is that they move transaction and smart
contract execution off-chain (to Layer 2). Layer 1 nodes can then
focus on tasks that require high decentralization but are
computationally lightweight, such as consensus.</p>
<p>While Smart Rollups are designed so that anyone can carry out
execution for any rollup, execution <em>does not need</em> to be widely
decentralized. That is because rollup operators are required to
regularly post receipts, <em>commitments</em>, to Layer 1 with the result of
the execution. These can be verified by anyone, and dishonest or
erroneous commitments can be challenged by honest actors to ensure integrity.</p>
<p>However, rollup operators must of course be able to receive
instructions from the rollup’s users to carry out any execution in the
first place.</p>
<p>One way is to include these instructions in Layer 1 blocks, which is
done on Tezos using the shared <a href="https://research-development.nomadic-labs.com/introducing-data-availability-committees.html#the-rollup-inbox">rollup
inbox</a>. This
ensures that the original transaction data is available for verifying
the execution done by rollup operators. But even when applying
compression techniques, Layer 1 block size will eventually become a
bottleneck. A rough estimate puts the maximum throughput using this
approach on Tezos at approximately 3400 <span class="caps">TPS</span><sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup> with current protocol
parameters (block size, block time, etc).</p>
<p>For handling millions of transactions per second (<span class="caps">TPS</span>) it’s necessary
to keep transaction data out of Layer 1 blocks. But once it is
off-chain, how can we trust it, and how can we be sure it is available
for others to verify the work of rollup operators?</p>
<p>This is known as the data-availability problem and is a widely
recognized issue among those trying to scale blockchains with Layer 2 solutions.</p>
<h2 id="data-availability-committees-are-already-here"><a class="toclink" href="#data-availability-committees-are-already-here">Data-availability committees are already here</a></h2>
<p>For achieving millions of <span class="caps">TPS</span> on Tezos we currently rely on <em>Data-Availability Committees</em>, or DACs for short.</p>
<p>A <span class="caps">DAC</span> is a group of data providers that host data for rollups
off-chain and make it available via the <a href="https://tezos.gitlab.io/alpha/smart_rollups.html#reveal-data-channel">reveal data
channel</a>,
an interface enabling Smart Rollups to access data external to the
Tezos blockchain. For a deeper dive into DACs, see <a href="https://research-development.nomadic-labs.com/introducing-data-availability-committees.html">this
article</a>.</p>
<p>The <span class="caps">DAC</span> can provide a high degree of security as long as a few <span class="caps">DAC</span>
participants are honest, with the tradeoff that a few dishonest
participants can limit the throughput of the rollup.</p>
<p>But ultimately the Tezos protocol does not control or monitor what
data is stored on a <span class="caps">DAC</span> and can provide no guarantees about the data
being available. As a result, using DACs involves some trust in its
members actually storing the data and providing it on request.</p>
<p>DACs remain practical for many applications, especially ones that
don’t require a high level of decentralization or guarantees of rollup
data being publicly available. This can be gaming, ticketing, or other
use cases where a centralizing actor is already involved. DACs are
also practical when private control of the data is required for
confidentiality or legal reasons.</p>
<p>But Tezos’ enshrined rollups are intended as an integrated,
decentralized scaling stack. Thus, the next step is to make sure
rollup data’s availability is guaranteed by the Tezos protocol itself.</p>
<h2 id="the-fully-decentralized-solution-a-data-availability-layer"><a class="toclink" href="#the-fully-decentralized-solution-a-data-availability-layer">The fully decentralized solution: A data-availability layer</a></h2>
<p>A <em>Data-Availability Layer</em>, or <span class="caps">DAL</span>, stores data and provides
guarantees about its availability by relying on Layer 1 consensus,
i.e. bakers.</p>
<p>It’s an independent peer-to-peer (<span class="caps">P2P</span>) network, running in parallel
with Tezos’ Layer 1, where data can be submitted and retrieved. Bakers
continuously monitor the <span class="caps">DAL</span> and <em>attest</em> on Layer 1 whether a given
piece of data is available on the <span class="caps">DAL</span>.</p>
<p>Unlike the <span class="caps">P2P</span> protocol employed by Layer 1, where each node receives
all data, the <span class="caps">P2P</span> protocol utilized by the <span class="caps">DAL</span> is designed such that
<span class="caps">DAL</span> nodes receive only part of the data. Each data “chunk” is
accompanied by an <a href="https://www.techtarget.com/searchstorage/definition/erasure-coding">erasure
code</a>,
which makes it possible to reconstruct the original data, even if
parts of the data are lost or never properly published.</p>
<p>In addition to attestations by bakers, <span class="caps">DAL</span> nodes perform data
availability sampling<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup>, a technique that ensures, with a high
probability, that the data is available while only downloading a small
portion of the entire data. This means that anyone can ascertain data
availability merely by running a node, without needing to trust bakers.</p>
<p>Overall, this approach is similar to what is proposed for Ethereum
under the name ‘danksharding’, and what is planned for Celestia. It
effectively circumvents the limitations inherent to Layer 1, massively
scaling bandwidth and storage capacity for the Tezos network, while
maintaining a high level of security and decentralization.</p>
<p>A quick overview of roles and responsibilities:</p>
<ul>
<li><strong>Adding data:</strong> Anyone can submit new data to the <span class="caps">DAL</span>, though
pre-approval by Layer 1 is required to deter spamming.</li>
<li><strong>Storing data:</strong> Anyone can contribute to storing data. The more
people contributing, the higher the resiliency and efficiency of the
<span class="caps">DAL</span>.</li>
<li><strong>Verifying availability:</strong> Bakers continuously publish
<em>attestations</em> on Layer 1, declaring the availability of the
data. Other <span class="caps">DAL</span> nodes perform data availability sampling.</li>
<li><strong>Retrieving data:</strong> Anyone can retrieve any data from the <span class="caps">DAL</span>.</li>
</ul>
<p>Smart Rollups were designed for future compatibility with a <span class="caps">DAL</span> and
will be able to leverage it as soon as it is activated through a
protocol upgrade.</p>
<h2 id="notes-on-infrastructure"><a class="toclink" href="#notes-on-infrastructure">Notes on infrastructure</a></h2>
<p>Given the different <span class="caps">P2P</span> protocol, we’ve decided to implement a
separate node for connecting to the <span class="caps">DAL</span> network. We aim to make the
command line interface and the configuration closely resemble those of
the Octez node.</p>
<p>It means that both rollup operators and bakers will need to run <span class="caps">DAL</span>
nodes, though based on feedback from the community, the setup with
separate <span class="caps">DAL</span> nodes may change in the future. Hardware recommendations
for participating in the <span class="caps">DAL</span> will be provided at a later stage, when
the <span class="caps">DAL</span> has been evaluated on test networks.</p>
<p>Bakers should be aware that <em>attesting</em> requires downloading the data,
which can be demanding in terms of bandwidth. Similarly to Layer 1
consensus, the amount of <span class="caps">DAL</span> attestations by a baker will be
proportional to their stake, and bandwidth requirements will therefore
also depend on their stake.</p>
<h2 id="status-and-roadmap"><a class="toclink" href="#status-and-roadmap">Status and roadmap</a></h2>
<p>An early version of the <span class="caps">DAL</span> is now available on the
<a href="https://teztnets.xyz/mondaynet-about">Mondaynet</a>
testnet. Concurrently, we are preparing the <span class="caps">DAL</span> for production,
including stress testing its <span class="caps">P2P</span> protocol.</p>
<p>The current <span class="caps">DAL</span> version does not have data-availability sampling
implemented, and participation is optional for testnet bakers. We
expect both to change before Mainnet release, which is estimated for
early 2024.</p>
<p>Those interested in the technical aspects of the <span class="caps">DAL</span> can explore our
<a href="https://hackmd.io/@p-cUv0l5RNaDKBCowZ0IzA/HJgFgSzpo/https%3A%2F%2Fhackmd.io%2FUQuA_59QRdOjU47fGM9CsQ">design
document</a>. Also,
more blog posts will follow, explaining design decisions and providing
in-depth technical descriptions.</p>
<p>We consider the <span class="caps">DAL</span> a ground-breaking innovation when it comes to
providing decentralized data-availability solutions. Upon activation
through a protocol upgrade, the <span class="caps">DAL</span> would solidify Tezos’ position as
a technical leader – and a blockchain that doesn’t compromise on decentralization.</p>
<!-- Footnotes themselves at the bottom. -->
<h2 id="notes"><a class="toclink" href="#notes">Notes</a></h2>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>Assuming a Layer 2 transaction takes up 10 Bytes of Layer 1 block
space. Roughly, we can put at most <span class="caps">512KB</span> of transactions in a
Layer 1 block, which is equivalent to 51.2K transactions. As
block time is 15 seconds, this means we can achieve a throughput
of about 3413 <span class="caps">TPS</span>. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>Data availability sampling is not implemented in the <span class="caps">DAL</span> version
currently available on test networks. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>Higher TPS and new rollup features: Nairobi upgrade is live!2023-06-24T02:30:00+02:002023-06-24T02:30:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2023-06-24:/Nairobi-is-live.html<p>Tezos has activated its fourteenth protocol upgrade, Nairobi.</p><p>On June 24 2023 00:07:10 <span class="caps">UTC</span>, the Tezos blockchain successfully
activated the Nairobi protocol upgrade at block
<a href="https://tzstats.com/3760129">#3,760,129</a>.</p>
<p>This 14th upgrade was jointly developed by Nomadic Labs, Marigold,
TriliTech, Oxhead Alpha, Tarides, DaiLambda, and Functori.</p>
<p>Included in Nairobi:</p>
<ul>
<li><a href="https://research-development.nomadic-labs.com/nairobi-announcement.html#increased-tps-thanks-to-a-finer-grained-gas-model">An up to 8x increase in
<span class="caps">TPS</span></a>
for transactions, smart contract calls, Smart Rollup maintenance
operations, and other <em>manager operations</em>, thanks to an improved
gas model for cryptographic signatures.</li>
<li><a href="https://research-development.nomadic-labs.com/nairobi-smart-rollups.html">New functionality for Smart
Rollups</a>,
including new host functions and new internal Layer 2 messages
allowing rollup kernels to sync with Tezos protocol upgrades.</li>
<li><a href="https://research-development.nomadic-labs.com/nairobi-announcement.html#renaming-endorsements-to-attestations">Renaming
endorsements</a>
to attestations.</li>
<li><a href="https://research-development.nomadic-labs.com/nairobi-announcement.html#fine-tuned-validation-pipelining-for-faster-consensus">Faster propagation of
pre-attestations</a>
to reach consensus earlier.</li>
</ul>
<p>For more details, see the <a href="https://research-development.nomadic-labs.com/nairobi-announcement.html">Nairobi announcement
post</a>. A
deeper technical description can be found in the protocol proposal’s
<a href="https://tezos.gitlab.io/nairobi/protocol.html">technical
documentation</a>, and a
complete list of changes is provided in <a href="https://tezos.gitlab.io/protocols/017_nairobi.html">Nairobi’s
changelog</a>.</p>
<h2 id="join-us-at-tezdev-2023"><a class="toclink" href="#join-us-at-tezdev-2023">Join us at TezDev 2023!</a></h2>
<p><a href="https://tezos.com/tez-dev/">TezDev</a> is back in Paris on July 21 for a
full day of innovation and collaboration, bringing together voices
from across the ecosystem. Join us to connect with a thriving
community, get inspired, and learn from experts on the latest and most
exciting developments on the Tezos blockchain!</p>
<p>This year’s focus is the
<a href="https://twitter.com/hashtag/RiseofRollups">#RiseofRollups</a>, with
experts from Nomadic Labs, Marigold, Trilitech, and Tezos Commons
charting the course for Layer 2 scaling on Tezos.</p>
<p>Key speakers include: </p>
<ul>
<li>Arthur Breitman, co-founder of Tezos.</li>
<li>Martin Lynge, <span class="caps">VP</span> of Gaming for Misfits Gaming and Block Born.</li>
<li>Vlad Horilyi, <span class="caps">COO</span> of Madfish Solutions.</li>
<li>Jean Schmitt, Lead Backend Developer at Ubisoft.</li>
</ul>
<p>In keynote speeches, Nomadic Labs’ Hadrien Zerah will cover adoption
strategies and value creation, and Tezos co-founder Arthur Breitman
will present an ambitious vision for the future of rollups on Tezos.</p>
<p>Throughout the day there will be workshops covering data-availability
solutions, how to develop Smart Rollup kernels, and the upcoming <span class="caps">EVM</span>
support on Tezos.</p>
<p>Additionally, panel sessions will unpack DeFi success stories on
Tezos, review the state of Layer 2s in 2023, and discuss universal interoperability.</p>
<p>… and, of course, there will be lots of networking opportunities in
the beautiful surroundings of La Fabrique Événementielle in the heart
of Paris.</p>
<h2 id="register-today"><a class="toclink" href="#register-today">Register today</a></h2>
<p><a href="https://tezos.us6.list-manage.com/track/click?u=80b9a27c332a234b4cac5c13b&id=7d20c36549&e=abe606c229">Register
here</a>
to secure your spot. Use promo code <code>Tez0sEarLy1</code> to get a €15
discount on your ticket. Early bird offer ends on June 30 so act now!</p>
<p>We look forward to seeing you!</p>Incident Report: Mumbai 2 user-activated protocol override2023-06-16T15:00:00+02:002023-06-16T15:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2023-06-16:/Mumbai2-incident-report.html<p>The updated Mumbai upgrade proposal, Mumbai 2, patched a vulnerability that could potentially halt block production on the Tezos network. No funds were at risk.</p><!-- Imported from
https://docs.google.com/document/d/1GINaHvSVpHTZ2091d3nuDJqDF_fnlO7ZXJS_3AYEF-g/edit#heading=h.lmwgmstsq1u3
-->
<p><strong><span class="caps">TL</span>;<span class="caps">DR</span> The updated Mumbai upgrade proposal, Mumbai 2, patched a
vulnerability that could potentially halt block production on the
Tezos network. No funds were at risk.</strong></p>
<p>On March 7th 2023, <a href="https://research-development.nomadic-labs.com/mumbai2-announcement.html">we announced Mumbai
2</a>,
a patched version of the Mumbai protocol proposal addressing a
liveness vulnerability witnessed on the Ghostnet test network. In this
report, we revisit the events as they occurred and the decisions taken
in response to this incident. But first, we provide a short summary:</p>
<ul>
<li>An <em>“inconsistent hash”</em> error was reported on
<a href="https://teztnets.xyz/ghostnet-about">Ghostnet</a> level
<a href="https://ghostnet.tzkt.io/2022087/operations">#2,022,087</a>.</li>
<li>Issue was tracked back to an inconsistency in the Tezos protocol
cache, which stored different representations of certain values
depending on node uptime for a deployed smart contract.</li>
<li><strong>Network was safe</strong>: at worst this issue would affect network
liveness (i.e. block production would stall or stop), but not lead
to an inconsistent ledger state.</li>
<li>Mumbai 2: A patched version of the Mumbai protocol proposal <a href="https://research-development.nomadic-labs.com/mumbai2-announcement.html">was
published on March
7th</a>.</li>
<li><a href="https://research-development.nomadic-labs.com/Mumbai-is-live.html">Mumbai 2 activated
successfully</a>
on Tezos Mainnet on block level #3,268,609.</li>
<li>There was no evidence of this issue being exploited (or attempted to
be exploited) on Ghostnet or Tezos Mainnet.</li>
</ul>
<h2 id="incident-discovery"><a class="toclink" href="#incident-discovery">Incident discovery</a></h2>
<p>On February 21st 2023, we observed that several nodes in the
<a href="https://teztnets.xyz/ghostnet-about">Ghostnet</a> test network reported
an <em>“inconsistent hash”</em> error message for block proposals for level
<a href="https://ghostnet.tzkt.io/2022087/operations">#2,022,087</a>.</p>
<p>Our investigation identified an issue in the way <a href="https://tezos.gitlab.io/michelson-reference/#type-lambda">Michelson
Lambdas</a> are
stored on the economic protocol’s cache, which manifested as a
divergence in the cache entry for a deployed contract.
<a href="https://ghostnet.tzkt.io/KT1Ja7Cq1HUTzmk1Qh8iERrEzp1LCjRXvqei">KT1Ja7Cq1HUTzmk1Qh8iERrEzp1LCjRXvqei</a>,
between different nodes.</p>
<p>This divergence was not the result of a bug in the smart contract nor
the Michelson interpreter, but rather on a difference in runtime
behavior that would lead to certain nodes storing an <em>“unoptimized”</em>,
human readable version of their argument in one case, versus an
optimized byte-based representation in the other, depending on their
uptime and recent activity.</p>
<h2 id="risk-assessment-and-mitigation"><a class="toclink" href="#risk-assessment-and-mitigation">Risk assessment and mitigation</a></h2>
<p>This issue threatened network liveness, as it risked dividing nodes
depending on the values stored in their protocol caches. If each side
accounted for more than one third of the attestation power but less
than two thirds, block production would halt. It’s worth highlighting
that under Tenderbake consensus rules, there was no risk of a
diverging network split where each fork progressed separately<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>.</p>
<p>In this worst case scenario, the ledger state would remain safe, but
not live.</p>
<p>Even so, it was quickly noted that should that scenario occur, there
would be a relatively straightforward path to recovery. It would
suffice to reboot enough nodes holding the unoptimized value in the
cache for them to be updated with the optimized value, and get the
network unstuck.</p>
<p>In spite of this, and the lack of evidence that there was indeed a
Byzantine motivation in the deployment of this contract, we still
decided to treat this issue as a 0-day vulnerability, and worked on
fixing this issue quickly and silently:</p>
<ul>
<li>The vulnerability had been witnessed on Ghostnet, a public test
network, and nothing prevented it from happening again on the same
test network nor on Mainnet – if an exploit was derived.</li>
<li>The information necessary to weaponize this bug was public, and
moreover didn’t require a very deep understanding of the core issue
– it needed only deploying and interacting with a similar contract
on Mainnet.</li>
</ul>
<p>We had also investigated various ways of further mitigating the
problem via an Octez shell update, but none were found to be
satisfactory. Thus, we had no option to address this issue at its
core, and modify the way the Tezos economic protocol interacts with
the cache:<em> when updating an entry, values should be always normalized
to the optimized, byte representation</em>.</p>
<p>We took the decision to deploy this fix<strong> only on</strong> <strong>Mumbai</strong>, after
considering the following:</p>
<ul>
<li>While this was a high risk threat, it was not a safety issue – no
funds were compromised, there was no risk of an inconsistent ledger
state –, but rather a liveness one: at worst, block production would
slow down or grind to a halt.</li>
<li>Even if this bug was indeed present in Lima on Tezos Mainnet, there
was no evidence that the contract triggering the incident was
deployed with an intent to attack the network, nor were there
consecutive attempts to exploit it – neither on Ghostnet nor Mainnet.</li>
<li>Mitigation was straightforward by rebooting affected nodes. If the
situation escalated (e.g. by a repeated exploit), we would still
have the option to deploy a patch for Lima on Mainnet as a
<em>user-activated</em> upgrade.</li>
</ul>
<p>After thorough testing and review, an updated protocol proposal,
Mumbai 2, <a href="https://research-development.nomadic-labs.com/mumbai2-announcement.html">was
announced</a>
on March 7th.</p>
<p>In hindsight, this decision seems to be the adequate one: the
user-activated protocol override was not controversial, and after the
original Mumbai proposal successfully passed the Promotion period
vote, <a href="https://research-development.nomadic-labs.com/Mumbai-is-live.html">Mumbai
2</a>
activated at block #3,268,609 on March 29th. Moreover, we did not
witness neither a repetition of the incident nor any attempt to
exploit it.</p>
<h2 id="moving-forward"><a class="toclink" href="#moving-forward">Moving forward</a></h2>
<p>Ideally, we would have discovered this issue on bleeding-edge test
networks like <a href="https://teztnets.xyz/dailynet-about">Dailynet</a> or
<a href="https://teztnets.xyz/mondaynet-about">Mondaynet</a>.</p>
<p>However, this requires being able to support environments closer to
Tezos Mainnet in test networks: more smart contracts and rollups
deployed, increased traffic, etc. Moving forward, we are working
towards increasing the capability to reproduce Mainnet conditions in
our test infrastructure. It is also imperative to increase community
participation in bleeding-edge test networks.</p>
<p>Tezos is constantly evolving, and a new protocol upgrade,
<a href="https://research-development.nomadic-labs.com/nairobi-announcement.html">Nairobi</a>,
is set to activate on Tezos Mainnet <a href="https://tzkt.io/3760129">around June
23rd</a>. Staying on top of the blockchain game
requires us to move fast. Yet, we don’t buy the part of the mantra
which requires breaking things – at least, not on Mainnet.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>Indeed when we said that adopting Tenderbake was <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html#nakamoto-is-live-classical-bft-is-safe">a trade-off of
between (more) safety and (less)
liveness</a>,
we had scenarios like this in mind: in the event of a network
split, it is not possible for both forks to advance as at most
one can reach sufficient attestation power. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Introducing Data Availability Committees2023-05-23T16:00:00+02:002023-05-23T16:00:00+02:00TriliTech and Nomadic Labstag:research-development.nomadic-labs.com,2023-05-23:/introducing-data-availability-committees.html<p>Data Availability Comittees enable storing transaction data for Smart Rollups off-chain, increasing the scalability of Tezos Layer 2.</p><p>A Data Availability Committee (<span class="caps">DAC</span>) is a solution for scaling the
transaction throughput of Tezos <a href="https://tezos.gitlab.io/alpha/smart_rollups.html">Smart
Rollups</a>. In
summary, a <span class="caps">DAC</span> enables storing transaction data for a smart rollup
off-chain. Rollup nodes retrieve the transaction payloads from the <span class="caps">DAC</span>
members and import them into their Smart Rollup virtual machines,
instead of retrieving them directly from Tezos blocks, and thus
circumvent the data limit imposed by Tezos block sizes. In this
article we’ll take a look at the infrastructure built by
<a href="https://www.trili.tech/">TriliTech</a> and
<a href="https://www.marigold.dev/">Marigold</a> engineers to support a <span class="caps">DAC</span>. But
before delving into the specifics, let’s first understand a bit better
how rollups work, to see how they fit in.</p>
<h2 id="smart-rollups"><a class="toclink" href="#smart-rollups">Smart Rollups</a></h2>
<p>With <a href="https://research-development.nomadic-labs.com/Mumbai-is-live.html">the activation of the Mumbai protocol
upgrade</a>
on Tezos, Smart Rollups are live on Mainnet. A Smart Rollup is an
application that is executed off-chain by one or many rollup nodes
that periodically submits a hash of the application state
on-chain. Anyone can run a smart rollup node and post commitments by
locking a bond of 10,000 tez. Furthermore, the Tezos economic protocol
provides a mechanism to challenge and disprove fraudulent
commitments. Since Smart Rollups execute outside of Layer 1, they have
the potential to massively scale the amount of computations involved
in running blockchain applications.</p>
<p>There are currently two data sources from which a Smart Rollup can
import messages to process: the protocol-wide rollup inbox and the
reveal data channel. We’ll cover both of them shortly, but before
doing so let’s outline the requirements we’d like to satisfy:</p>
<ul>
<li><em>Integrity</em>: The data imported into a Smart Rollup must be
verifiably correct, that is, it is possible to prove that it has not
been tampered with.</li>
<li><em>Availability</em>: A Smart Rollup must be able to retrieve any message
addressed to it. Not being able to do so could hinder the progress
of a rollup at best, and could allow a dishonest party to take
control of the rollup at worst.</li>
</ul>
<h2 id="the-rollup-inbox"><a class="toclink" href="#the-rollup-inbox">The Rollup Inbox</a></h2>
<p>The rollup inbox is stored in the Tezos blockchain’s context. Users
add messages to the rollup inbox <a href="https://tezos.gitlab.io/active/blocks_ops.html#manager-operations-mumbai">via a dedicated manager
operation</a>. Rollup
nodes then download blocks from the Tezos network to retrieve the
contents of the inbox.</p>
<p>The protocol ensures that the data published through the rollup inbox
is well-defined, that is there can be no disputes about what messages
consist of or in which order they were received. Data availability is
also guaranteed by being able to download blocks through a node
participating in the Tezos network. However, this method sacrifices
scalability since the bandwidth of the rollup inbox is limited by the
size of a block (currently 500 <span class="caps">KB</span>) and the minimum time it takes to
bake a new block (15 seconds at the time of writing). As a result, the
bandwidth of the rollup inbox is restricted to around 33 <span class="caps">KB</span>/s, which
is further shared among all active rollups.</p>
<h3 id="the-reveal-data-channel"><a class="toclink" href="#the-reveal-data-channel">The Reveal Data Channel</a></h3>
<p>To overcome the scalability limitations of the rollup inbox, smart
rollups offer an additional source for importing data — via the
reveal data channel. This allows <a href="https://tezos.gitlab.io/nairobi/smart_rollups.html#developing-wasm-kernels">rollup
kernels</a>
— the application logic of a rollup — to request from the rollup
node the
<em><a href="https://river.com/learn/terms/p/preimage#:~:text=A%20preimage%20is%20the%20data,hash%20of%20a%20public%20key">preimage</a></em>
of a given hash. Such a preimage is called a <em>page</em> and has a size
limit of <span class="caps">4KB</span>. The rollup node is expected to provide pages in a fixed
location known as the <em>reveal data directory</em>. There is no limit on
the number of pages a rollup can request at each Tezos level, ensuring
scalability. The integrity of the data is enforced since a preimage is
requested through its hash. However, unlike the Smart Rollup’s inbox,
the_ reveal data channel_ does not guarantee availability of the
data. That is, there are no assurances that a rollup node will have
the page corresponding to a given hash when it’s requested. This is
precisely the problem addressed by Data Availability Committees.</p>
<h2 id="data-availability-committees"><a class="toclink" href="#data-availability-committees">Data Availability Committees</a></h2>
<p>A <span class="caps">DAC</span> consists of a group of parties that commit to storing copies of
input data and keeping the data available upon request. Our
implementation of DACs:</p>
<ul>
<li>
<p>Provides the infrastructure needed to send, distribute and store
data of arbitrary size among the <span class="caps">DAC</span> members.</p>
</li>
<li>
<p>Enables rollup nodes to download payloads from a <span class="caps">DAC</span> and to populate
the reveal data directory.</p>
</li>
<li>
<p>Defines a communication pattern that can be used by Smart Rollups to
import the data stored by committee members, provided that a
sufficient number of committee members have signed their commitment
to make a copy of the data available.</p>
</li>
</ul>
<p>It’s important to note that the <span class="caps">DAC</span> stack is external to the Tezos
economic protocol. That is, the Tezos Layer 1 is not aware of any
<span class="caps">DAC</span>. The relationship between DACs and Smart Rollups is one-to-many. A
<span class="caps">DAC</span> can serve multiple rollups while a single rollup must use at most
one <span class="caps">DAC</span>.</p>
<p>Here’s a high-level description of the workflow:</p>
<p><img src="/images/dac/dac_workflow.png" alt="Tezos DAC architecture workflow" width="100%"></p>
<ol>
<li>The user sends the payload to a <span class="caps">DAC</span> (1a) and waits for a
certificate with a sufficient number of signatures (1b), as
determined by the rollup kernel (the application logic of a
particular rollup).</li>
<li>The certificate, which is small in size (approximately 140 bytes),
is posted to the rollup inbox as a Layer 1 message (2a) and will
eventually be downloaded by the rollup node (2b).</li>
<li>The rollup kernel imports the certificate contained in the rollup
inbox, and verifies that it contains valid signatures of several
committee members. It is the responsibility of the rollup kernel to
define the minimum number of signatures required by a certificate
to be considered valid,</li>
<li>If the certificate is deemed valid, the rollup kernel will request
to import the pages of the original payload to the rollup node. The
rollup node downloads those pages from the <span class="caps">DAC</span> infrastructure (4a)
before importing them into the kernel (4b).</li>
</ol>
<p>The rollup kernel must implement the logic to determine if a <span class="caps">DAC</span>
certificate is valid, and to request the original payload by importing
the corresponding pages through the reveal data channel.</p>
<h2 id="deploying-a-dac-committee"><a class="toclink" href="#deploying-a-dac-committee">Deploying a <span class="caps">DAC</span> committee</a></h2>
<p>Following is a brief description of how to set up and deploy a <span class="caps">DAC</span>
committee for serving data for a Smart Rollup. Note that the software
has not yet been released, but it still can be tested on
<a href="https://teztnets.xyz/mondaynet-about">Mondaynet</a> and
<a href="https://teztnets.xyz/nairobinet-about">Nairobinet</a>, after building
Octez <a href="https://tezos.gitlab.io/introduction/howtoget.html#compiling-with-make">from
sources</a>.</p>
<p>For reference, this blog post uses the current <a href="https://gitlab.com/tezos/tezos/-/tree/182746e806ff8bb4f0deaa1e83f152c579552e41">Tezos repository
master
branch</a>
at the time of writing. We plan to publish all the <span class="caps">DAC</span> binaries with
one of the coming Octez releases.</p>
<p>After building from source, users will find two experimental executables:</p>
<ul>
<li><code>./octez-dac-node</code> (the <span class="caps">DAC</span> node)</li>
<li><code>./octez-dac-client</code> (the <span class="caps">DAC</span> client)</li>
</ul>
<p>The <code>octez-dac-node</code> executable can be used to set up a new committee
or track an existing one. The <code>octez-dac-client</code> can be used to send
payloads to the <span class="caps">DAC</span> for storage, and to retrieve certificates signed
from the data availability committee.</p>
<p>To set up a <span class="caps">DAC</span>, several inter-connected instances of the <span class="caps">DAC</span> node
must be executed. In particular, the <span class="caps">DAC</span> node supports three modes of
operations: coordinator mode, committee member mode, and observer
mode, described next.</p>
<p><img src="/images/dac/dac_infra.png" alt="Tezos DAC roles orchestration" width="90%"></p>
<h3 id="coordinator"><a class="toclink" href="#coordinator">Coordinator</a></h3>
<p>The Coordinator acts as a gateway between the clients of the <span class="caps">DAC</span> and
the other <span class="caps">DAC</span> nodes. It is responsible for receiving a payload and
splitting it into pages of 4KBs each — the maximum size of a preimage
that can be imported into a rollup — and forwarding the resulting
pages to other nodes. It is also responsible for providing <span class="caps">DAC</span> clients
with data availability certificates. A <span class="caps">DAC</span> node running in coordinator
mode must have access to the public keys of the committee members. A
coordinator can be configured with the following command:</p>
<div class="highlight"><pre><span></span><code>./octez-dac-node configure as coordinator with data availability committee members $TZ4_PUBLIC_KEYS --data-dir $DATA_DIR --reveal-data-dir $REVEAL_DATA_DIR
</code></pre></div>
<p>where:</p>
<ul>
<li><code>$TZ4_PUBLIC_KEYS</code> is a list of <span class="caps">BLS</span> aggregate account (tz4 accounts)
public keys.</li>
<li><code>$DATA_DIR</code> is the directory containing the persisted store of the
<span class="caps">DAC</span> node instance. This argument is optional and will default to
<code>~/.octez-dac-node</code> when missing. It is suggested to give it an
explicit value in case multiple dac nodes run on the same host.</li>
<li><code>$REVEAL_DATA_DIR</code> is a separate directory where payloads are
stored. This argument is also optional.</li>
</ul>
<p>Once it has been configured, the coordinator can be run with:</p>
<div class="highlight"><pre><span></span><code>./octez-dac-node run --data-dir $DATA_DIR
</code></pre></div>
<p>where <code>$DATA_DIR</code> is the same as for the configuration command</p>
<h3 id="committee-member"><a class="toclink" href="#committee-member">Committee Member</a></h3>
<p>A committee member receives pages from the coordinator and stores them
on disk. Once all the pages for the original payload are received, the
committee member sends a cryptographic signature to the coordinator to
confirm its commitment to storing the data and making it available to
external entities upon request. The coordinator collects these
signatures and includes them in the data availability certificate for
the payload.</p>
<p>To connect with the coordinator, a committee member node needs to use
the following command for configuration:</p>
<div class="highlight"><pre><span></span><code>./octez-dac-node configure as committee member with coordinator $COORDINATOR_RPC_ADDR and signer $TZ4_ADDRESS --data-dir $DATA_DIR --reveal-data-dir $REVEAL_DATA_DIR
</code></pre></div>
<p>where,</p>
<ul>
<li><code>$COORDINATOR_RPC_ADDR</code> is the <span class="caps">RPC</span> address of the coordinator node,
in the format <code>{host}:{port}</code>;</li>
<li><code>$TZ4_ADDRESS</code> is the address of the tz4 account that will be used
to sign commitments to the availability of payloads; and,</li>
<li><code>$DATA_DIR</code> and<code>$REVEAL_DATA_DIR</code> serve the same function as in the
coordinator node, but should have different values from the
coordinator node if the two run on the same machine.</li>
</ul>
<h3 id="observer"><a class="toclink" href="#observer">Observer</a></h3>
<p>Similar to a committee member, an observer node also receives pages
from the coordinator and stores them on disk. If the observer is run
on the same host machine as a rollup node, and its reveal data
directory is set to the same one as on the rollup node, it becomes
responsible for providing the pages corresponding to the input
payload. To configure an observer you can run the following command:</p>
<div class="highlight"><pre><span></span><code>./octez-dac-node configure as observer with coordinator $COORDINATOR_RPC_ADDR and committee member rpc addresses $COMMITTEE_MEMBE RPC_ADDRESSES --data-dir $DATA_DIR --reveal-data-dir $REVEAL_DATA_DIR
</code></pre></div>
<p>where,</p>
<ul>
<li><code>$COMMITTEE_MEMBER_RPC_ADDRESSES</code> is the list of the <span class="caps">RPC</span> addresses
of the committee member nodes, in the format <code>{host}:{port}</code>.</li>
</ul>
<h2 id="retrieving-a-dac-certificate"><a class="toclink" href="#retrieving-a-dac-certificate">Retrieving a <span class="caps">DAC</span> certificate</a></h2>
<p>Once the <span class="caps">DAC</span> infrastructure has been set up, users can request the
committee members to store a payload of arbitrary size via the
<code>octez-dac-client</code> command, by running the following:</p>
<div class="highlight"><pre><span></span><code><span class="o">./</span><span class="n">octez</span><span class="o">-</span><span class="n">dac</span><span class="o">-</span><span class="n">client</span><span class="w"> </span><span class="n">send</span><span class="w"> </span><span class="n">payload</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">coordinator</span><span class="w"> </span><span class="o">$</span><span class="n">COORDINATOR_RPC_ADDR</span><span class="w"> </span><span class="n">with</span><span class="w"> </span><span class="n">content</span><span class="w"> </span><span class="o">$</span><span class="n">PAYLOAD</span><span class="w"> </span><span class="o">--</span><span class="n">wait</span><span class="o">-</span><span class="k">for</span><span class="o">-</span><span class="n">threshold</span><span class="w"> </span><span class="o">$</span><span class="n">THRESHOLD</span>
</code></pre></div>
<p>where,</p>
<ul>
<li><code>$COORDINATOR_RPC_ADDR</code> is the address of the <span class="caps">DAC</span> coordinator, in
the form <code>{host_ip}:{port}</code></li>
<li><code>$PAYLOAD</code> is the hex-encoded payload that committee members will store</li>
<li><code>$THRESHOLD</code> is the minimum number of committee members that must
commit to make the data available, before the command returns.</li>
</ul>
<p>Upon executing the command, a hex-encoded data availability
certificate is returned, with a size of approximately 140 bytes. This
certificate can be posted to the global rollup inbox and will
eventually be processed by the rollup kernel.</p>
<h2 id="next-steps"><a class="toclink" href="#next-steps">Next Steps</a></h2>
<p>Moving forward, we aim to enhance DACs to ensure seamless integration
with Smart Rollups. Some of our planned improvements include:</p>
<ul>
<li>
<p>Introducing new <span class="caps">DAC</span>-related functions to the <a href="https://gitlab.com/tezos/tezos/-/tree/master/src/kernel_sdk">Rollup Kernel
<span class="caps">SDK</span></a>.</p>
</li>
<li>
<p>Allowing committee members to commit to data availability for a set
period of time.</p>
</li>
<li>
<p>Provide in-depth tutorials demonstrating how to write a rollup
kernel that utilizes DACs.</p>
</li>
</ul>
<p>We welcome all community feedback on their experience with DACs to
help inform our future efforts, so don’t hesitate to join the
conversation at
<a href="https://join.slack.com/t/tezos-dev/shared_invite/zt-1g80svzg7-klZaVglekzSD~aOA54rV_w">#scoru-ecosystem</a>
on the Tezos Dev Slack workspace.</p>What’s Cooking for Smart Rollups in Nairobi?2023-05-11T15:00:00+02:002023-05-11T15:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2023-05-11:/nairobi-smart-rollups.html<p>An upgrade that makes life easier for Smart Rollup developers.</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: The Nairobi protocol upgrade brings new features and improvements to Smart Rollups that will make life easier for developers.</strong></p>
<p>Mumbai has only just found its way to Mainnet, and a new protocol amendment is already being proposed to the community: Nairobi.</p>
<p>As mentioned in the <a href="https://research-development.nomadic-labs.com/nairobi-announcement.html">Nairobi announcement</a>, the upgrade brings several quality of life improvements for developers of Smart Rollups. This blog post is an introduction to these new features. It is aimed at developers who are already familiar with Smart Rollups, but includes context and links to existing documentation for less versed readers. </p>
<p>First and foremost, it is important to understand that <em>every new feature covered by this blogpost will benefit every Smart Rollup on Mainnet</em>, including those deployed prior to activation of Nairobi. Hence you can already deploy your Smart Rollup today, and benefit from the new features once Nairobi reaches Mainnet (though you will need to upgrade your kernel for some of them).</p>
<p>This “free upgrade” in Nairobi is a good example of the advantages that come with Smart Rollups being <em>enshrined </em>– i.e. <a href="https://research-development.nomadic-labs.com/smart-rollups-are-coming.html#enshrined-what">part of the Tezos protocol</a> itself. It is also a first re-affirmation of Tezos core developers’ commitment to continuously maintain and improve Smart Rollups.</p>
<h2 id="the-protocol_migration-internal-message"><a class="toclink" href="#the-protocol_migration-internal-message">The Protocol_migration Internal Message</a></h2>
<p><a href="http://tezos.gitlab.io/nairobi/smart_rollups.html#internal-messages">Internal messages</a> are a particular kind of message sent to the Smart Rollups shared inbox by the Tezos protocol itself. In Mumbai, there are four kinds of internal messages:</p>
<ul>
<li><code>Start_of_level</code> is the very first message of the shared inbox in every Tezos level.</li>
<li><code>Info_per_level</code> comes just after <code>Start_of_level</code> and provides some information about the Layer 1 progress (namely, the block hash and timestamp of the previous Tezos block).</li>
<li><code>Transfer</code> messages are injected by the protocol when a Michelson smart contract performs a contract call targeting a Smart Rollup.</li>
<li><code>End_of_level</code> is the very last message of the shared inbox for any Tezos level.</li>
</ul>
<p>Nairobi introduces a new kind of internal message, called <code>Protocol_migration</code>. This message will be injected by the protocol after <code>Info_per_level</code> for the very first block of a given Layer 1 protocol. This way, Smart Rollup kernels can be made aware of new protocol activations. This opens new possibilities, like a kernel taking advantage of new features as soon as they are available.</p>
<h2 id="the-typed_transactions_batch-outbox-message"><a class="toclink" href="#the-typed_transactions_batch-outbox-message">The Typed_transactions_batch Outbox Message</a></h2>
<p>In Mumbai, Smart Rollup kernels can interact with Layer 1 thanks to their outbox. More precisely, they can send operation batches which can be executed in the Layer 1 as soon as the commitment containing them is <a href="https://tezos.gitlab.io/alpha/smart_rollups.html#commitments">cemented</a>, i.e. can no longer be refuted. </p>
<p>However, these batches have a potential flaw, in the sense that they are untyped. This is an issue, because some Micheline expressions can potentially have several valid types. Worst case scenario, a kernel could potentially be tricked into withdrawing tickets by mistake. Even if this cannot be used to forge tickets on the Layer 1 (which is well protected against tickets forgery), this could potentially allow attackers to drain a vulnerable rollup. </p>
<p>To prevent such a scenario, Nairobi introduces a new kind of outbox message which allows the kernel to specify the types of the batched transactions it wants to execute on Layer 1.</p>
<h2 id="the-first-wasm-pvm-upgrade"><a class="toclink" href="#the-first-wasm-pvm-upgrade">The First <span class="caps">WASM</span> <span class="caps">PVM</span> Upgrade</a></h2>
<p>In our communications about Smart Rollups we have continuously emphasized how the runtime of Smart Rollups, the <em><span class="caps">WASM</span> <a href="https://research-development.nomadic-labs.com/next-generation-rollups.html#the-proof-generating-virtual-machine">Proof-generating Virtual Machine</a></em> (<span class="caps">PVM</span>), allows for installed kernels to be upgraded.</p>
<p>But that isn’t all. In Nairobi we demonstrate that the <span class="caps">WASM</span> <span class="caps">PVM</span> itself is also upgradable.</p>
<p>A kernel can inspect the version of the <span class="caps">WASM</span> <span class="caps">PVM</span> executing it by looking at its durable storage, more precisely under the key <code>/readonly/wasm_version</code>. The version name of the <span class="caps">PVM</span> released in Mumbai is <code>2.0.0</code>.</p>
<p>This can be verified using an up-to-date [octez-smart-rollup-wasm-debugger], which also provides an option to choose which <span class="caps">WASM</span> <span class="caps">PVM</span> to use when debugging (with the <code>-p</code> option).</p>
<p>Nairobi introduces the first revision of the <span class="caps">WASM</span> <span class="caps">PVM</span>, version <code>2.0.0-r1</code>, which is key to the new features contained in the protocol upgrade. Every Smart Rollup originated in Mumbai will see their <span class="caps">PVM</span> upgrade automatically at the beginning of the first block of Nairobi, should the protocol amendment be adopted by the community.</p>
<p>The new version is fully backwards compatible with the interactive fraud proof system introduced in Mumbai: refutation games started before the activation of Nairobi will not be affected by the presence of these new features.</p>
<h2 id="stack-size-limit"><a class="toclink" href="#stack-size-limit">Stack Size Limit</a></h2>
<p>In version <code>2.0.0</code> of the <span class="caps">PVM</span>, the stack of a kernel is limited to 300 stack frames. This value was inherited from the reference <span class="caps">WASM</span> interpreter, but early adopters of Smart Rollups quickly reported it to be fairly limited, and hard to satisfy. To this day, <code>octez-smart-rollup-wasm-debugger</code> is still lacking the feature to tell you how many frames at most have been created by <code>kernel_run</code> invocations. Besides, Wasmer (the standard execution engine used for Smart Rollups when no proof is required) does not limit its own stack in terms of frames count, but rather in terms of memory usage.</p>
<p>As a consequence, in version <code>2.0.0-r1</code> the stack size limit of the <span class="caps">WASM</span> <span class="caps">PVM</span> has been bumped significantly to ensure that as long as Wasmer is able to execute a kernel, the <span class="caps">WASM</span> <span class="caps">PVM</span> can execute it too. This should make the lives of kernel developers easier.</p>
<h2 id="two-new-host-functions"><a class="toclink" href="#two-new-host-functions">Two New Host Functions</a></h2>
<p>Finally, version <code>2.0.0-r1</code> of the <span class="caps">WASM</span> <span class="caps">PVM</span> introduces two new host functions:</p>
<ul>
<li><code>store_create</code> lets you preallocate large values in the durable storage almost for free, ticks wise. It is analogous to calling<a href="https://man7.org/linux/man-pages/man2/ftruncate.2.html"> truncate</a> to an empty file in most file systems.</li>
<li><code>store_delete_value</code> is the counterpart of <code>store_delete</code>, restricted to the value under a given key. The durable storage allows a given key to be both a value and a directory, and <code>store_delete</code> deletes both.</li>
</ul>
<p>The following figure summarizes the difference between the two. Consider a durable storage which contains three values, at <code>/foo</code>, <code>/foo/bar</code> and <code>/foobar</code>. Calling <code>store_delete</code> with <code>/foo</code> as its argument also removes <code>/foo/bar</code>, while <code>store_delete_value</code> will leave <code>/foo/bar</code> untouched.</p>
<p><img src="https://research-development.nomadic-labs.com/images/SR-foobar-diagram.png" alt= “Diagram” width="100%"></p>
<h2 id="warning-against-store_get_nth_key"><a class="toclink" href="#warning-against-store_get_nth_key">Warning against store_get_nth_key</a></h2>
<p>Please note that the <code>store_get_nth_key</code> host function is now <em>considered harmful</em>. While it has not been removed from the <span class="caps">WASM</span> <span class="caps">PVM</span> set of host functions, we advise kernel developers to not use it, as it has been discovered that its behavior is potentially incompatible with the refutation game.</p>
<p>Kernels which do not use this host function are safe, and we see no indication of this issue affecting the other host functions.</p>On the future of validity rollups on Tezos2023-05-04T17:00:00+02:002023-05-04T17:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2023-05-04:/future-validity-rollups.html<p>We are combining optimistic rollups and validity rollups in a single product.</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: Validity rollups currently suffer from a “scaling trilemma” that calls for a strategic shift in how they are integrated into Tezos. Instead of offering both optimistic and validity rollups, we will combine them in a single product.</strong></p>
<p>Validity rollups (aka. zk-rollups) are all the rage, and we would like to update the community on our work to bring this technology to Tezos.</p>
<p>As you may know from <a href="https://research-development.nomadic-labs.com/mumbai-preview.html">previous communications</a>, our implementation is referred to as Epoxy, and an early version is enabled on Mondaynet. In order to test the system we have also developed <code>epoxy-tx</code>, a transactional rollup capable of handling Tezos’ tickets.</p>
<p>It is the result of two years of R&D by our <a href="https://research-development.nomadic-labs.com/files/cryptography.html">cryptography team</a> and has given us great insights into the usefulness and applicability of Zero Knowledge Proofs, but also into the challenges and limitations involved. This has led us to draw some conclusions about this technology that may be surprising for some. </p>
<p>In short, we believe that validity technology won’t achieve general compatibility and high throughput at a reasonable cost for at least a few years. Not just on Tezos, but in general.</p>
<p>In this blog post we lay out our perspective on the current state of validity rollups, based on our research and conclusions. And we present an exciting strategic shift in how we aim to integrate this technology into Tezos in a way that counterbalances its intrinsic limitations.</p>
<h2 id="optimistic-rollups-vs-validity-rollups"><a class="toclink" href="#optimistic-rollups-vs-validity-rollups">Optimistic rollups vs. validity rollups</a></h2>
<p>The <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html">scaling roadmap</a> for Tezos, published in early 2022, focused on two rollup technologies: optimistic rollups and validity rollups.</p>
<p>The Tezos variant of <strong>optimistic rollups</strong>, <a href="https://research-development.nomadic-labs.com/smart-rollups-are-coming.html">Smart Rollups</a>, launched with the Mumbai upgrade. With optimistic rollups, the work of rollup operators is treated as honest by default, hence the “optimistic” element. However, if there is foul play, an operator’s fraud can be refuted by another operator by posting a proof demonstrating the wrong-doing within a two-week period. </p>
<p>Because proofs are only produced in case of a dispute, hardware requirements for rollup operators are moderate even with high throughput and complex operations. You can run any virtual machine – on Tezos, Smart Rollups offer a <a href="https://tezos.gitlab.io/alpha/smart_rollups.html#developing-wasm-kernels"><span class="caps">WASM</span> execution environment</a>. And it takes just one honest operator to ensure the integrity of the rollup.</p>
<p>The main drawback is the two-week dispute period. Until this period has passed, transactions in the rollup can’t be considered final, and when withdrawing assets from the rollup to Layer 1, assets are only released after expiration of the dispute period. Additionally, all incoming transaction data needs to be kept publicly available for verification during this period, though <a href="https://research-development.nomadic-labs.com/smart-rollups-are-coming.html#ready-for-data-availability-solutions">solutions for this</a> are underway.</p>
<p><strong>Validity rollups </strong>work differently. A small proof is posted with every commitment guaranteeing the correctness of the operator’s work. The proof is small, lightweight and can be easily verified by anyone. Hence, a validity rollup can be run by a single operator without concerns about the honesty about that operator’s work. Foul play is automatically rejected by the protocol itself, and no other actor needs to keep track of the rollup to guarantee security.</p>
<p>Additionally, the technology enables the state and operations of the rollup to (optionally) be completely hidden from the main chain. They can even be hidden from the users of the service, e.g. each user knows their balance and transactions but not anyone else’s. Only rollup operators have full access to the data of the rollup in order to produce the proof.</p>
<p>Third, providing a proof with every commitment means that validity rollups have instant finality. This greatly simplifies the security analysis and the implementation of applications when compared with optimistic rollups.</p>
<p>So, is this the solution that solves everything? Unfortunately not. </p>
<h2 id="the-challenges-with-validity-rollups"><a class="toclink" href="#the-challenges-with-validity-rollups">The challenges with validity rollups</a></h2>
<p>Validity rollups currently have their own significant drawbacks – some of which we believe deserve <em>a bit more</em> attention in the general promotion of them as a scaling solution. </p>
<p>These drawbacks are a couple of interconnected challenges that we, and everyone else working on this technology, are currently faced with.</p>
<p><strong>Challenge #1: Proofs are very expensive.</strong> Creating <span class="caps">SNARK</span> proofs – which must be done with every commitment – requires a considerable amount of computing power. This makes running validity rollups very expensive and puts a limit on the complexity of operations that can be expressed (see challenge #2). For example, support of standard cryptography, such as <a href="https://en.wikipedia.org/wiki/EdDSA#Ed25519">ed25519</a> signatures (used for ‘tz1’ accounts) and <a href="https://en.wikipedia.org/wiki/BLAKE_(hash_function)#BLAKE2">Blake2b</a> hashes, is currently difficult to achieve at high throughput. New cryptographic primitives for use in validity rollups are being developed, but this again requires that existing infrastructure is adapted.</p>
<p>In our efforts to address this challenge, we have developed <a href="https://gitlab.com/nomadic-labs/cryptography/privacy-team/">aPlonK</a>, an advanced proving system tailored to Tezos, which uses a<a href="https://eprint.iacr.org/2022/1352"> novel technique</a> for efficient proof aggregation. Essentially, it reduces the proof size and verification time when multiple statements are proven in a batch. It contains a language to describe circuits, and a prover that enables proof-generation to be distributed over a cluster of machines to achieve, in theory, unlimited scalability. The limiting factor being the data-center bill.</p>
<p>But regardless of computing power (and funds), even our best prover, fully optimized and parallelized, cannot go faster than an optimistic rollup operator, which by default does not produce any proof, but simply executes the program and posts a commitment. As long as there is at least one honest participant continuously validating rollup activity, security is ensured and in a much more cost-efficient way than using <span class="caps">SNARK</span> proofs.</p>
<p><strong>Challenge #2: Limited compatibility.</strong> The ability to execute arbitrary code is a crucial feature for a Layer 2 solution. This not only enables compatibility with existing smart contracts programmed for blockchain ecosystems (such as <span class="caps">EVM</span> and Michelson), but also enables the Layer 2 solution to become a distributed backend for more conventional development environments.</p>
<p>However, validity rollups rely on so-called Succinct Non-interactive Arguments of Knowledge (SNARKs) for their proofs. For this to work, all statements must be translated into <em>circuits</em> – a set of mathematical equations that the proving systems can process. In effect, every smart contract or dApp must be ported into what is essentially another programming language. </p>
<p>This also brings us back to challenge #1: current proving technology doesn’t support sufficiently advanced circuits to interpret the execution of existing smart contracts directly, if cost of computation is to be kept reasonable. Hence, general compatibility is currently prohibitively expensive at the throughput necessary to make validity rollups relevant as scaling solutions. Again, this is true not just for Epoxy, but for validity tech in general.</p>
<p><strong>Challenge #3: Fragmentation in tooling.</strong> Due to the use of circuits for the proofs, a whole new stack of tooling, SDKs, wallet integration and other kinds of infrastructure must be created. While this in itself may not be the biggest challenge, having parallel stacks for Layer 1, optimistic rollups and validity rollups introduces a level of fragmentation which we believe will become problematic. We believe there are better ways, which we will go into further below.</p>
<h2 id="a-current-validity-trilemma"><a class="toclink" href="#a-current-validity-trilemma">A (current) validity trilemma</a></h2>
<p>The above can be illustrated as a “trilemma” of validity rollups. The three desired properties of validity rollups are</p>
<ul>
<li><strong>Compatibility</strong> – existing dApps and programs can be executed in the rollup</li>
<li><strong>Reasonable cost</strong> – requirements for computing power are realistic</li>
<li><strong>High throughput</strong> – proofs can cover many operations and still be generated fast enough for blockchain purposes</li>
</ul>
<p>Smart Rollups give you all three properties – at the cost of longer finality – while the current state of validity rollup tech allows you to only pick two.</p>
<p>If you want high compatibility and reasonable cost, throughput will be too low for practical purposes. If you go for high compatibility and high throughput, it will become incredibly expensive to run a rollup – we’re talking <em>massive</em> data centers. And if you prioritize reasonable cost of operation and high throughput, the complexity of operations that can be processed will be severely limited. For example, epoxy-tx, our rollup for <a href="https://tezos.gitlab.io/alpha/tickets.html">Tezos tickets</a> transactions, has high throughput and low cost relative to other validity designs, but is limited to, well, transactions.</p>
<p>Of course, a lot of work is currently going into reducing the required computational resources. However, our own experiments and extensive review of current research into this by various projects in the industry lead us to conclude that this trilemma will remain relevant for at least a couple of years – possibly longer.</p>
<p>A brief overview of currently available validity rollups and their approach:</p>
<ul>
<li><strong>Optimizing for throughput (and cost):</strong> ZkSync Lite, dYdX, Loopring, Immutable X follow the same philosophy as Tezos’ Epoxy – relying on application specific <span class="caps">ZK</span> circuits to get good scalability (in the order of 1000 transactions per second). Building those circuits, however, is an expensive and delicate task, and compatibility is limited.</li>
<li><strong>Optimized for compatibility (and cost)</strong>: ZkVM projects, such as ZkSync Era and Polygon ZkEVM, go for a compatibility-first approach. Even though this technology is promising, we’ve <a href="https://l2beat.com/scaling/activity">yet to see them achieve high throughput</a>. </li>
</ul>
<h2 id="so-what-about-epoxy"><a class="toclink" href="#so-what-about-epoxy">So, what about Epoxy?</a></h2>
<p>The question is then: what does this conclusion mean for our work? </p>
<p>To answer that question, it is important to make clear what we have built. We have referred to our exploration into validity rollups on Tezos as Epoxy, but it actually consists of two parts: a prover and a connecting framework. </p>
<p>Our prover is called aPlonk and includes a language for describing circuits. The prover is the ‘engine’ of a validity rollup and by far the most important element. And it’s the part we have spent most resources developing. What we call Epoxy is the framework, the glue, that connects this proving system to the Tezos blockchain.</p>
<p>Based on our conclusions presented above, we have decided to go a different route than launching Epoxy as a product competing with Smart Rollups. Not because we don’t believe in a bright future for validity rollups – far from it. In fact, we are excited to be able to give a sneak peek into a strategic shift that we believe will benefit everyone using rollups on Tezos.</p>
<p>We are essentially taking the Epoxy prototype apart and re-purposing the engine and other parts in what we believe is a revolutionary new product.</p>
<h2 id="the-hybrid-approach"><a class="toclink" href="#the-hybrid-approach">The hybrid approach</a></h2>
<p>In Smart Rollups, we already have a high-compatibility and high-throughput solution at a low cost, but with longer finality. Tooling is developing rapidly and soon vast Smart Rollup-based infrastructure will be built out on Tezos. </p>
<p>Launching validity rollups as a product competing with Smart Rollups, but with different tooling and the above mentioned trade-offs, is not the best approach. </p>
<p>What is the better approach? Upgrading Smart Rollups with validity tech! Think instant finality for a higher fee. Or standard transactions with short finality, and longer finality for more complex operations. Or new confidentiality features.</p>
<p>This ‘hybrid’ approach has several advantages</p>
<ul>
<li><strong>Cementing Smart Rollup longevity:</strong> Users of Smart Rollups can be confident that their toolchains and infrastructure will remain relevant for the foreseeable future.</li>
<li><strong>Complimentary features:</strong> Validity tech can be implemented where it makes sense and used depending on the needs and priorities of a given rollup. Optimistic rollup tech covers everything else.</li>
<li><strong>Gradual implementation:</strong> As validity tech matures, the balance between optimistic and validity elements can be adjusted through kernel upgrades after new features are introduced in Tezos protocol upgrades. In other words, existing Smart Rollups can evolve as validity technology evolves.</li>
</ul>
<p>Make no mistake: In the long term, we see a bright future for validity rollups. But our analysis tells us that the technology just isn’t there yet for them to be competitive with optimistic rollups. Again, this is not specific for Epoxy, but for validity rollups in general. We believe that a gradual implementation, resting on the solid foundation of Smart Rollups, is the right solution for Tezos for the years to come.</p>
<p>The hybrid design for Smart Rollups is ongoing R&D, but we will soon be able to release more details. We look forward to embarking on this exciting journey in cooperation with the Tezos ecosystem and community!</p>Labelled type parameters in OCaml2023-04-25T14:00:00+02:002023-04-25T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2023-04-25:/labelled-type-parameters-in-ocaml.html<p>You can have labelled type parameters in OCaml, here is how!</p><p>The OCaml language allows for
<a href="https://v2.ocaml.org/manual/lablexamples.html">labelled parameters in function definitions and calls</a>.
This is a pleasant feature of the language which can be used to make the code
self-documenting. For example consider the difference between the <code>blit</code>
functions in the <code>Stdlib.String</code> and <code>Stdlib.StringLabels</code>.</p>
<div class="highlight"><pre><span></span><code><span class="k">val</span> <span class="n">blit</span> <span class="o">:</span> <span class="kt">string</span> <span class="o">-></span> <span class="kt">int</span> <span class="o">-></span> <span class="n">bytes</span> <span class="o">-></span> <span class="kt">int</span> <span class="o">-></span> <span class="kt">int</span> <span class="o">-></span> <span class="kt">unit</span>
<span class="k">val</span> <span class="n">blit</span> <span class="o">:</span> <span class="n">src</span><span class="o">:</span><span class="kt">string</span> <span class="o">-></span> <span class="n">src_pos</span><span class="o">:</span><span class="kt">int</span> <span class="o">-></span> <span class="n">dst</span><span class="o">:</span><span class="n">bytes</span> <span class="o">-></span> <span class="n">dst_pos</span><span class="o">:</span><span class="kt">int</span> <span class="o">-></span> <span class="n">len</span><span class="o">:</span><span class="kt">int</span> <span class="o">-></span> <span class="kt">unit</span>
</code></pre></div>
<p>In the second version, the meaning of each parameter is easier to
remember and their order is not a source of confusion. And when the
function is called, the order of the parameters can be changed:</p>
<div class="highlight"><pre><span></span><code><span class="n">blit</span> <span class="o">~</span><span class="n">len</span><span class="o">:</span><span class="mi">1024</span> <span class="o">~</span><span class="n">src</span><span class="o">:</span><span class="n">s</span> <span class="o">~</span><span class="n">src_pos</span><span class="o">:</span><span class="mi">0</span> <span class="o">~</span><span class="n">dst</span><span class="o">:</span><span class="n">b</span> <span class="o">~</span><span class="n">dst_pos</span><span class="o">:</span><span class="mi">0</span>
</code></pre></div>
<p>The OCaml language supports type constructors. But the type parameters cannot be
labelled. The meaning and order of each parameter can become a source of confusion.
E.g., <a href="https://v2.ocaml.org/api/Stdlib.html#TYPEformat6">with the <code>format</code> types from the OCaml <code>Stdlib</code></a>:</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span><span class="o">,</span> <span class="k">'</span><span class="n">b</span><span class="o">,</span> <span class="k">'</span><span class="n">c</span><span class="o">,</span> <span class="k">'</span><span class="n">d</span><span class="o">,</span> <span class="k">'</span><span class="n">e</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">format6</span> <span class="o">=</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span><span class="o">,</span> <span class="k">'</span><span class="n">b</span><span class="o">,</span> <span class="k">'</span><span class="n">c</span><span class="o">,</span> <span class="k">'</span><span class="n">d</span><span class="o">,</span> <span class="k">'</span><span class="n">e</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="nn">CamlinternalFormatBasics</span><span class="p">.</span><span class="n">format6</span>
<span class="k">type</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span><span class="o">,</span> <span class="k">'</span><span class="n">b</span><span class="o">,</span> <span class="k">'</span><span class="n">c</span><span class="o">,</span> <span class="k">'</span><span class="n">d</span><span class="o">)</span> <span class="n">format4</span> <span class="o">=</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span><span class="o">,</span> <span class="k">'</span><span class="n">b</span><span class="o">,</span> <span class="k">'</span><span class="n">c</span><span class="o">,</span> <span class="k">'</span><span class="n">c</span><span class="o">,</span> <span class="k">'</span><span class="n">c</span><span class="o">,</span> <span class="k">'</span><span class="n">d</span><span class="o">)</span> <span class="n">format6</span>
<span class="k">type</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span><span class="o">,</span> <span class="k">'</span><span class="n">b</span><span class="o">,</span> <span class="k">'</span><span class="n">c</span><span class="o">)</span> <span class="n">format</span> <span class="o">=</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span><span class="o">,</span> <span class="k">'</span><span class="n">b</span><span class="o">,</span> <span class="k">'</span><span class="n">c</span><span class="o">,</span> <span class="k">'</span><span class="n">c</span><span class="o">)</span> <span class="n">format4</span>
</code></pre></div>
<p>Well it turns out there is a way to label type parameters and we are about to
show you how!</p>
<h2 id="orthogonal-features"><a class="toclink" href="#orthogonal-features">Orthogonal features</a></h2>
<p>One of the strength of the OCaml language is how separate orthogonal
features tend to combine predictably.</p>
<p>It turns out OCaml’s type system has some orthogonal features that,
taken together, let you label your type parameters.</p>
<ul>
<li><p>Type constructors with type parameters: Type constructors allow
to parametrise a whole family of types over some other types. For
example, the <code>Stdlib.Result</code> module defines a type:</p>
<div class="sourceCode" id="cb4"><pre
class="sourceCode ocaml"><code class="sourceCode ocaml"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a><span class="kw">type</span> ('a, 'e) t = Ok <span class="kw">of</span> 'a | Error <span class="kw">of</span> 'e</span></code></pre></div>
<p>We call <code>'a</code> and <code>'e</code> the <em>type
parameters</em> and <code>t</code> the <em>type constructor</em>
because, given actual types for the parameters (say <code>int</code> and
<code>string</code>) the application <code>(int, string) t</code>
construct an actual type.</p>
<p>Note that the parameter name chosen to represent the error type uses
the mnemonic <code>'e</code>. However, when you encounter an instance
such as <code>(int, string) t</code>, there are no indication which
parameter is which. (The <code>Result.t</code> type is common enough
through the OCaml ecosystem that this is not generally a problem; we are
just setting up a small manageable example.)</p></li>
<li><p>Object types: Types which describe objects. These type mentions
the publicly available methods of an object. Importantly for our
purpose, they mention those by names and are equivalent regardless of
the order they appear in.</p>
<div class="sourceCode" id="cb5"><pre
class="sourceCode ocaml"><code class="sourceCode ocaml"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="kw">type</span> o = < get_x: <span class="dt">int</span>; get_y: <span class="dt">int</span> ></span></code></pre></div></li>
<li><p>Type parameter constraints: Narrows down the possible
instantiations of a type parameter. This is a rarely used feature but it
is handy on occasions (e.g., when passing a module that is monomorphic
through and through to a functor which expects one with polymorphism).</p>
<div class="sourceCode" id="cb6"><pre
class="sourceCode ocaml"><code class="sourceCode ocaml"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="kw">type</span> 'a t = 'a <span class="dt">list</span></span>
<span id="cb6-2"><a href="#cb6-2" aria-hidden="true" tabindex="-1"></a> <span class="kw">constraint</span> 'a = <span class="dt">int</span></span></code></pre></div></li>
</ul>
<p>Now putting all three features together, we can define labelled type parameters:</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="k">'</span><span class="n">p</span> <span class="n">res</span> <span class="o">=</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span><span class="o">,</span> <span class="k">'</span><span class="n">e</span><span class="o">)</span> <span class="nn">Result</span><span class="p">.</span><span class="n">t</span>
<span class="k">constraint</span> <span class="k">'</span><span class="n">p</span> <span class="o">=</span> <span class="o"><</span> <span class="n">ok</span><span class="o">:</span> <span class="k">'</span><span class="n">a</span><span class="o">;</span> <span class="n">error</span><span class="o">:</span> <span class="k">'</span><span class="n">e</span> <span class="o">></span>
</code></pre></div>
<p>This is a bit circumvoluted, but essentially the type constructor <code>res</code>
is a one-parameter alias for the two-parameter type constructor
<code>Result.t</code>. And the type parameter of <code>res</code> is constrained to be an
object type with two methods. The method names serve as our parameter labels:</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="n">catch_exceptions</span>
<span class="o">:</span> <span class="o">(</span><span class="kt">unit</span> <span class="o">-></span> <span class="k">'</span><span class="n">a</span><span class="o">)</span> <span class="o">-></span> <span class="o"><</span> <span class="n">ok</span><span class="o">:</span> <span class="k">'</span><span class="n">a</span><span class="o">;</span> <span class="n">error</span><span class="o">:</span> <span class="kt">string</span> <span class="o">></span> <span class="n">res</span>
<span class="o">=</span> <span class="k">fun</span> <span class="n">f</span> <span class="o">-></span>
<span class="k">match</span> <span class="n">f</span> <span class="bp">()</span> <span class="k">with</span>
<span class="o">|</span> <span class="n">v</span> <span class="o">-></span> <span class="nc">Ok</span> <span class="n">v</span>
<span class="o">|</span> <span class="n">exc</span> <span class="o">-></span> <span class="nc">Error</span> <span class="o">(</span><span class="nn">Printexc</span><span class="p">.</span><span class="n">to_string</span> <span class="n">exc</span><span class="o">)</span>
</code></pre></div>
<h2 id="a-real-world-use-case"><a class="toclink" href="#a-real-world-use-case">A real-world use-case</a></h2>
<p>The example above is somewhat artificial: the type constructor
<code>Result.t</code> has only two parameters and is widespread in the OCaml
ecosystem. But the idea can be used for real-world use-cases. In fact
it is!</p>
<p>In the <a href="gitlab.com/tezos/tezos">Octez Tezos suite</a>, one of the type
constructor has six parameters. The type constructor is for <span class="caps">RPC</span>
services and the parameters correspond to:</p>
<ul>
<li>the type of allowed <span class="caps">HTTP</span> methods,</li>
<li>the type of parameters in the <span class="caps">URL</span> (think <code>/block/<level>/operations</code>),</li>
<li>the type of parameters in the <span class="caps">URL</span> but differently this time because of reasons irrelevant to the blog post at hand,</li>
<li>the type of parameters in the query section of the <span class="caps">URL</span> (think <code>/version?format=<format></code>),</li>
<li>the type of parameters in the body of the request, and</li>
<li>the type of the returned value.</li>
</ul>
<p>So roughly, there is a type</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="o">(</span><span class="k">'</span><span class="n">meth</span><span class="o">,</span> <span class="k">'</span><span class="n">prefix</span><span class="o">,</span> <span class="k">'</span><span class="n">params</span><span class="o">,</span> <span class="k">'</span><span class="n">query</span><span class="o">,</span> <span class="k">'</span><span class="n">input</span><span class="o">,</span> <span class="k">'</span><span class="n">output</span><span class="o">)</span> <span class="n">service</span> <span class="o">=</span> <span class="err">…</span>
</code></pre></div>
<p>The details are not too important for this blog post; the important
part is that this number of type parameters is cumbersome. To make the
code more readable (not more concise, but more readable), we
<a href="https://gitlab.com/tezos/tezos/-/blob/v16-release/src/lib_dal_node_services/services.mli#L34">introduce a type alias</a>
using the labelled type parameter technique above:</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="k">'</span><span class="n">rpc</span> <span class="n">service</span> <span class="o">=</span>
<span class="o">(</span><span class="k">'</span><span class="n">meth</span><span class="o">,</span> <span class="k">'</span><span class="n">prefix</span><span class="o">,</span> <span class="k">'</span><span class="n">params</span><span class="o">,</span> <span class="k">'</span><span class="n">query</span><span class="o">,</span> <span class="k">'</span><span class="n">input</span><span class="o">,</span> <span class="k">'</span><span class="n">output</span><span class="o">)</span> <span class="nn">Tezos_rpc</span><span class="p">.</span><span class="nn">Service</span><span class="p">.</span><span class="n">service</span>
<span class="k">constraint</span>
<span class="k">'</span><span class="n">rpc</span> <span class="o">=</span>
<span class="o"><</span> <span class="n">meth</span> <span class="o">:</span> <span class="k">'</span><span class="n">meth</span>
<span class="o">;</span> <span class="n">prefix</span> <span class="o">:</span> <span class="k">'</span><span class="n">prefix</span>
<span class="o">;</span> <span class="n">params</span> <span class="o">:</span> <span class="k">'</span><span class="n">params</span>
<span class="o">;</span> <span class="n">query</span> <span class="o">:</span> <span class="k">'</span><span class="n">query</span>
<span class="o">;</span> <span class="n">input</span> <span class="o">:</span> <span class="k">'</span><span class="n">input</span>
<span class="o">;</span> <span class="n">output</span> <span class="o">:</span> <span class="k">'</span><span class="n">output</span> <span class="o">></span>
</code></pre></div>
<p>With this alias in place we use concrete types based on labels rather
than position:</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="n">post_commitment</span> <span class="o">:</span>
<span class="o"><</span> <span class="n">meth</span> <span class="o">:</span> <span class="o">[`</span><span class="nc">POST</span><span class="o">]</span>
<span class="o">;</span> <span class="n">input</span> <span class="o">:</span> <span class="nn">Cryptobox</span><span class="p">.</span><span class="n">slot</span>
<span class="o">;</span> <span class="n">output</span> <span class="o">:</span> <span class="nn">Cryptobox</span><span class="p">.</span><span class="n">commitment</span>
<span class="o">;</span> <span class="n">prefix</span> <span class="o">:</span> <span class="kt">unit</span>
<span class="o">;</span> <span class="n">params</span> <span class="o">:</span> <span class="kt">unit</span>
<span class="o">;</span> <span class="n">query</span> <span class="o">:</span> <span class="kt">unit</span> <span class="o">></span>
<span class="n">service</span> <span class="o">=</span>
<span class="err">…</span>
</code></pre></div>
<h2 id="without-an-alias"><a class="toclink" href="#without-an-alias">Without an alias</a></h2>
<p>The combination of feature presented above only works as an alias of
an existing type constructor with unlabelled parameters. This is the
actual use-case we had in the Octez project because the <code>service</code> type
comes from an external library.</p>
<p>But you can eschew the alias using GADTs. Or rather <span class="caps">GADT</span> syntax. For example:</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="o">_</span> <span class="n">either</span> <span class="o">=</span>
<span class="o">|</span> <span class="nc">Left</span> <span class="o">:</span> <span class="k">'</span><span class="n">a</span> <span class="o">-></span> <span class="o"><</span><span class="n">left</span><span class="o">:</span> <span class="k">'</span><span class="n">a</span><span class="o">;</span> <span class="n">right</span><span class="o">:</span> <span class="k">'</span><span class="n">b</span><span class="o">></span> <span class="n">either</span>
<span class="o">|</span> <span class="nc">Right</span> <span class="o">:</span> <span class="k">'</span><span class="n">b</span> <span class="o">-></span> <span class="o"><</span><span class="n">left</span><span class="o">:</span> <span class="k">'</span><span class="n">a</span><span class="o">;</span> <span class="n">right</span><span class="o">:</span> <span class="k">'</span><span class="n">b</span><span class="o">></span> <span class="n">either</span>
</code></pre></div>
<h2 id="the-importance-of-orthogonal-features"><a class="toclink" href="#the-importance-of-orthogonal-features">The importance of orthogonal features</a></h2>
<p>It could be tempting to suggest that labelled type parameters should be
added to OCaml as a native feature. This could help with the syntax,
with the compiler error messages, and in a few other ways. However,
adding such a feature to the language increases the maintenance cost
of the compiler: the feature needs to be added, tested, documented,
and maintained through releases. It requires time. Time better spent
adding features which are unavailable, even by combining existing features.</p>
<p>OCaml’s strength is not measured as the sum of its features but as ways in which
these features combine together.</p>Announcing Nairobi, Tezos’ 14th protocol upgrade proposal2023-04-13T14:00:00+02:002023-04-13T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2023-04-13:/nairobi-announcement.html<p>Nairobi brings higher Layer 1 <span class="caps">TPS</span>, improved consensus broadcast, and new features for Smart Rollups.
Nairobi brings higher Layer 1 <span class="caps">TPS</span>, improved consensus broadcast, and new features for Smart Rollups.</p><p><em>This is a joint post from Nomadic Labs, Marigold, TriliTech, Oxhead
Alpha, Tarides, DaiLambda, <span class="amp">&</span> Functori.</em></p>
<p>Following the successful activation of the <a href="https://research-development.nomadic-labs.com/Mumbai-is-live.html">Mumbai protocol
upgrade</a>
on March 29th, we are happy to unveil our latest Tezos protocol
proposal, <strong>Nairobi</strong>.</p>
<p>As usual, Nairobi’s “true name” is its hash,
<code>PtNairobiyssHuh87hEhfVBGCVrK3WnS8Z2FT4ymB5tAa4r1nQf</code>.</p>
<p>The Nairobi protocol proposal contains several updates and
improvements to the Tezos economic protocol, the most prominent being:</p>
<ul>
<li>An up to 8x increase in <span class="caps">TPS</span> for certain manager operations
(including tez transfers and smart contract calls).</li>
<li>Improved gas model for signatures to reflect the cost of different curves.</li>
<li>Renaming endorsements to <em>attestations</em>.</li>
<li>Faster propagation of pre-attestations to reach consensus earlier.</li>
<li>New <a href="http://tezos.gitlab.io/alpha/smart_rollups.html#host-functions">host
functions</a>
for Smart Rollups, and new <a href="https://tezos.gitlab.io/alpha/smart_rollups.html#internal-messages">internal Layer 2
messages</a>
allowing rollup kernels to be aware of Tezos protocol upgrades.</li>
</ul>
<p>In this article, we give a preview into the improvements described
above. More in-depth descriptions can be found in <a href="https://tezos.gitlab.io/nairobi/protocol.html">Nairobi’s technical
documentation</a>. This
protocol proposal includes further minor improvements and other
changes – a complete list is provided in the protocol proposal’s
<a href="https://tezos.gitlab.io/protocols/017_nairobi.html">Changelog</a>.</p>
<h2 id="increased-tps-thanks-to-a-finer-grained-gas-model"><a class="toclink" href="#increased-tps-thanks-to-a-finer-grained-gas-model">Increased <span class="caps">TPS</span> thanks to a finer-grained gas model</a></h2>
<p>Currently, the gas cost of signature verification for manager
operations is a flat-tax constant, which over-approximates the
required resources. With Nairobi, their cost depends on the
cryptographic curve used and the size of the payload.</p>
<p>For instance, the gas cost of checking the signature of a <code>tz1</code> or
implicit account will be significantly smaller than those for a <code>tz3</code>,
in order to better reflect the difference in computational costs
associated with signature verification when using the <code>ed25519</code> (tz1)
and <code>p256</code> (tz3) cryptographic curves. See <a href="https://tezos.gitlab.io/protocols/017_nairobi.html#gas-improvements">this
entry</a>
in Nairobi’s Changelog for further detail (including breaking changes).</p>
<p>Thanks to this finer-grained accounting of the gas costs for signature
verification, the maximum number of <em><a href="https://tezos.gitlab.io/nairobi/blocks_ops.html#manager-operations">manager
operations</a></em>
(transactions, smart contract calls, Smart Rollups maintenance
operations, etc.) that can be included in a block has increased by a
factor of about 8.</p>
<p>Consequently, we observe a similar 8x increase in performance for
certain common operations, such as basic tez transactions between
<code>tz1</code> or <code>tz2</code> accounts<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>. For smart contract calls the number will
be lower, and operations involving ‘tz3’ accounts will see no increase
due to their more computationally demanding signature scheme.</p>
<h2 id="renaming-endorsements-to-attestations"><a class="toclink" href="#renaming-endorsements-to-attestations">Renaming Endorsements to Attestations</a></h2>
<p>The term “endorsement” might incorrectly convey the idea that bakers
(validators) approve of the contents of a block – that is, of the
transactions and other operations chosen by the payload producer –
even though in reality they are merely <em>attesting</em> that a valid block
has been produced. Moreover, ongoing work towards a <a href="https://research-development.nomadic-labs.com/smart-rollups-are-coming.html#ready-for-data-availability-solutions">Data Availability
Layer</a>
will change the behavior of consensus operations in a way that makes
the term “endorsements” less fitting.</p>
<p>We propose thus to rename endorsements into <em>“attestations”</em> – a more
precise term, which reflects better past, present, and potentially
future semantics of these consensus operations.</p>
<p>With this protocol proposal, we begin this process by launching the
deprecation of the <code>endorsing_rights</code> <span class="caps">RPC</span> endpoint, introducing
<code>attestation_rights</code> as a replacement. This symbolic step is not
breaking, as both old and new names would be available if Nairobi is
activated on Mainnet. In future protocol proposals, we intend to go
beyond symbolic steps though, and may have to introduce breaking
changes in <span class="caps">API</span> names – which will be communicated with ample notice.</p>
<h2 id="fine-tuned-validation-pipelining-for-faster-consensus"><a class="toclink" href="#fine-tuned-validation-pipelining-for-faster-consensus">Fine-tuned validation pipelining for faster consensus</a></h2>
<p>In order to keep Tezos consensus both live and fast, it is paramount
to ensure consensus operations are validated and propagated swiftly –
especially after Mumbai <a href="https://research-development.nomadic-labs.com/mumbai-preview.html#pipelining-is-fully-deployed-giving-us-15-second-block-times">halved block time from 30 to 15
seconds</a>
on Mainnet. To this end, this protocol proposal fine-tunes its Mempool
module, which implements the business logic used by the Octez node’s
to validate operations in the <em>mempool</em>.</p>
<p>With Nairobi, <a href="https://tezos.gitlab.io/shell/prevalidation.html">an Octez node’s
prevalidator</a> would
initialize mempool validation faster, and it would moreover accept
consensus operations that are <em>slightly in the future</em> or branched in
<em>cousin</em> blocks<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup>. As a result, the node can immediately propagate
pre-attestations for blocks which have just been validated but which
have not yet been applied. Consequently, bakers participating <a href="https://tezos.gitlab.io/nairobi/consensus.html#overview">in a
consensus
committee</a>
would be able to communicate their votes in the <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html#levels-are-composed-of-rounds">pre-attestation
voting
phase</a>
faster, which should lead to reaching consensus earlier on block proposals.</p>
<h2 id="continuous-improvement-of-smart-rollups"><a class="toclink" href="#continuous-improvement-of-smart-rollups">Continuous Improvement of Smart Rollups</a></h2>
<p>Tezos is the constantly evolving blockchain so it is only fitting that
Smart Rollups, its general-purpose Layer 2 solution, follow the same path.</p>
<p>This new protocol amendment, if deployed on Mainnet, will enable
additional <a href="http://tezos.gitlab.io/alpha/smart_rollups.html#host-functions">host
functions</a>
for all Smart Rollups. Nairobi also introduces a new kind of <a href="https://tezos.gitlab.io/alpha/smart_rollups.html#internal-messages">Layer 2
internal
message</a>
which allows to broadcast to all originated rollups the activation of
new Tezos protocol upgrades on Layer 1.</p>
<h2 id="buckle-up-nairobinet-test-network-will-launch-soon"><a class="toclink" href="#buckle-up-nairobinet-test-network-will-launch-soon">Buckle up, ‘Nairobinet’ test network will launch soon</a></h2>
<p>Note that if Nairobi is voted in by the community, upgrading to Octez
v17.0 (or later) will be necessary for participating in consensus.</p>
<p>In order to allow the community to start testing the Nairobi proposal
as soon as possible, a release candidate for Octez v17.0 will be
published in the coming days, and a dedicated protocol test network
<code>Nairobinet</code> is also scheduled to begin soon. More information about
this test network will be available on
<a href="https://teztnets.xyz/">https://teztnets.xyz/</a>.</p>
<p>Read more about Tezos test networks <a href="https://teztnets.xyz/">here</a>, and
don’t hesitate to reach out in the <a href="https://tezos-dev.slack.com/">Tezos Developer
Slack</a> or in the <a href="https://discord.com/invite/5v5QVKjvc2">Tezos
Discord</a> if you need help
getting started.</p>
<!-- Footnotes themselves at the bottom. -->
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>These observations arise from our experiments using the <span class="caps">TPS</span>
benchmark tool distributed with the Tezos protocol and the Octez
suite. See this earlier <a href="https://research-development.nomadic-labs.com/tps-evaluation.html">blog
post</a>
for further detail on how to replicate this experiment. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>Unpacking the jargon: operations in Tezos are <em>branched</em> (that
is, they reference) an earlier block hash. By <em>“slightly in the
future”</em>, we mean consensus operations not branched in the
predecessor of the target block (also known as the <em>“grandfather
block”</em>) but rather on other close descendants. Then, a <em>“cousin</em>
<em>block”</em> is a block who shares a grandfather with another block –
usually the successor of a block proposal for a different round
than the current mempool head. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>Smart Rollups and 15 second blocks: Mumbai upgrade is live!2023-03-29T19:00:00+02:002023-03-29T19:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2023-03-29:/Mumbai-is-live.html<p>Tezos has activated its thirteenth protocol upgrade, Mumbai.</p><p>On March 29 2023 16:40:29 <span class="caps">UTC</span>, the Tezos blockchain successfully
upgraded by activating the Mumbai proposal at block
<a href="https://tzstats.com/3268609">#3,268,609</a></p>
<p>This 13th Tezos protocol upgrade was jointly developed by Nomadic
Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda <span class="amp">&</span> Functori.</p>
<p>Included in Mumbai:</p>
<ul>
<li>
<p><a href="https://research-development.nomadic-labs.com/mumbai-preview.html#smart-rollups-activated">Smart Rollups are live on
Mainnet</a>:
With Smart Rollups, anyone can deploy decentralized WebAssembly
applications with dedicated computational and networking resources.</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/mumbai-preview.html#pipelining-is-fully-deployed-giving-us-15-second-block-times">Block time reduced to 15 seconds</a>: With improved pipelining validation fully deployed, block propagation times are significantly reduced, allowing for minimal block time to be halved to 15 seconds.</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/mumbai-preview.html#other-improvements">Ticket transfers between user accounts</a>: In Mumbai, <a href="https://tezos.gitlab.io/mumbai/tickets.html">Tezos tickets</a> can be transferred between user accounts (aka. implicit accounts) and not just to/from smart contracts and rollups.</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/mumbai-preview.html#other-improvements'">RPCs for ticket balances</a>: Two new <span class="caps">RPC</span> endpoints were added to improve the visibility of ticket ownership. <code>all_ticket_balances</code> returns a complete list of tickets owned by a given contract. And <code>ticket_balance</code> returns the given-contract’s balance of the ticket with specified ticketer, content type, and content.</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/mumbai-preview.html#other-improvements">New Michelson operations</a>: Michelson opcodes <code>AND</code>, <code>OR</code>, <code>XOR</code>, <code>NOT</code>, <code>LSL</code> and <code>LSR</code> were extended to support <a href="https://gitlab.com/tezos/tezos/-/merge_requests/6055">logical operations on bytes</a>, similar to those on nat. And an opcode was added to <a href="https://gitlab.com/tezos/tezos/-/merge_requests/6681">convert between bytes and nat</a> values.</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/mumbai-preview.html#epoxy-validity-rollups-on-mondaynet-testnet">A peek at validity rollups</a>: Our exploration into validity rollup (aka zk-rollup) technology, Epoxy, arrives on the Mondaynet testnet.</p>
</li>
</ul>
<p>For more details, see the <a href="https://research-development.nomadic-labs.com/mumbai-preview.html">Mumbai preview
post</a>.</p>
<p>A deeper technical description can be found in the protocol proposal’s
<a href="https://tezos.gitlab.io/mumbai/protocol.html">technical
documentation</a>, and a
complete list of changes is provided in <a href="https://tezos.gitlab.io/protocols/016_mumbai.html">Mumbai’s
changelog</a>.</p>
<h2 id="get-started-with-smart-rollups"><a class="toclink" href="#get-started-with-smart-rollups">Get started with Smart Rollups</a></h2>
<p>Smart Rollups are the backbone of <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html">Tezos’ scaling
strategy</a>
and hence instrumental in reaching our goal of <a href="https://research-development.nomadic-labs.com/smart-rollups-are-coming.html">1 million transactions
per
second</a>
in 2023.</p>
<p>But Smart Rollups are much more than that. They enable <a href="https://research-development.nomadic-labs.com/next-generation-rollups.html">entirely news
way of building decentralized
applications</a>
on Tezos, using any programming language that compiles to <span class="caps">WASM</span>, such
as Rust, C/C++, Go and Python.</p>
<p>The scalability and flexibility makes Smart Rollups the perfect
solution for any organization looking for a future-proof blockchain
solution. Don’t hesitate to reach out to Nomadic Labs’ <a href="https://adoption-support.nomadic-labs.com/contact/">adoption and
support team</a> if
you have questions regarding this.</p>
<p>For builders and tinkerers we recommend the <a href="https://tezos-dev.slack.com/">Tezos Developer
Slack</a> or the <a href="https://discord.com/invite/5v5QVKjvc2">Tezos
Discord</a> if you need help
getting started, as well as these community resources:</p>
<ul>
<li>
<p><a href="https://tezos.gitlab.io/alpha/smart_rollups.html">Smart Rollups</a>
(Tezos Technical Documentation)</p>
</li>
<li>
<p><a href="https://news.tezoscommons.org/setting-up-a-tezos-smart-rollup-in-5-steps-af62ed75a684">Setting up a Tezos Smart Rollup in 5 steps</a> (Tezos Commons)</p>
</li>
<li>
<p><a href="https://www.marigold.dev/post/how-to-write-a-rollup-kernel">How to Write a Rollup Kernel</a> (Marigold)</p>
</li>
<li>
<p><a href="https://www.marigold.dev/post/how-to-deploy-a-kernel-bigger-than-24kb">How to Deploy a Kernel Bigger Than 24kB</a> (Marigold)</p>
</li>
</ul>
<p>We are excited to be part of this major step in Tezos’ evolution, and
look forward to seeing Smart Rollups deployed in all varieties over
the coming months!</p>Liveness vulnerability found: A patched Mumbai proposal is available2023-03-07T16:00:00+01:002023-03-07T16:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2023-03-07:/mumbai2-announcement.html<p>We are proposing a patched Mumbai protocol upgrade to address a liveness vulnerability.</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: We are proposing a patched Mumbai protocol upgrade to address
a liveness vulnerability.</strong></p>
<p>We have discovered a vulnerability which could affect the Tezos
network’s liveness but not its safety. In other words, the
vulnerability could slow or halt the network, but not put funds at risk.</p>
<p>We have investigated various ways of mitigating the problem via an
<a href="https://tezos.gitlab.io/shell/shell.html">Octez shell</a> update
(everything but the protocol), but none were found to be satisfactory,
and a patch for the protocol itself is required.</p>
<p>We have taken the step of producing a patched version of the Mumbai
protocol upgrade proposal that addresses this vulnerability. The hash
of Mumbai 2, the patched version of Mumbai, is:
<code>PtMumbai2TmsJHNGRkD8v8YDbtao7BLUC3wjASn1inAKLFCjaH1</code>.</p>
<p>As the Mumbai proposal is currently in the <a href="https://www.tezosagora.org/period/92">Promotion
period</a> of the governance cycle,
there are two scenarios going forward:</p>
<ul>
<li>
<p><strong>User activated protocol override:</strong> The community votes <strong>Yay </strong>to
Mumbai in the current voting period. All nodes running v16.0 of
Octez will activate the patched version, <code>PtMumbai2</code>, instead of
<code>PtMumbaii</code>. This is what happened with the patched
<a href="https://research-development.nomadic-labs.com/mainnet-release-to-patch-babylon.html">Babylon</a>,
<a href="https://research-development.nomadic-labs.com/important-critical-patch-to-tickets-in-edo.html">Edo</a>,
and <a href="https://www.marigold.dev/post/announcing-hangzhou2">Hangzhou</a> proposals.</p>
</li>
<li>
<p><strong>Restarting the governance cycle:</strong> The community votes <strong>Nay</strong> to
Mumbai in the current voting period, and the governance process
“reverts” to a new proposal period. A patched Mumbai protocol will
then be proposed and must undergo a full governance cycle. This
should last about 2.5 months assuming approval in every voting period.</p>
</li>
</ul>
<p>A new release candidate, <a href="https://tezos.gitlab.io/releases/version-16.html#version-16-0-rc3">Octez
v16.0~rc3</a>,
has just been published, and v16.0 is expected shortly after. In
addition to the Mumbai 2 protocol, this release candidate also
includes performance improvements in the baker that we consider
necessary for the block time reduction <a href="https://research-development.nomadic-labs.com/mumbai-preview.html#pipelining-is-fully-deployed-giving-us-15-second-block-times">coming with
Mumbai</a>.</p>
<h2 id="important-octez-v160-will-be-required-for-mumbai"><a class="toclink" href="#important-octez-v160-will-be-required-for-mumbai">Important: Octez v16.0 will be required for Mumbai</a></h2>
<p>As stated in <a href="https://research-development.nomadic-labs.com/mumbai-announcement.html">a previous
blogpost</a>,
upgrading to v16.0 (or later) is necessary to participate in consensus
once Mumbai is activated, independent from the discussion above.</p>
<p>We remind the community that bakers representing at least 2/3 of the
total stake must be running the same protocol version for the chain to
stay live, due to the requirements of the
<a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">Tenderbake</a>
consensus algorithm.</p>
<p>The Mumbainet test network will be restarted shortly to instead use
the Mumbai 2 protocol. Joining the new rebooted test network will
require Octez v16.0~rc3, or the upcoming stable release. Please check
<a href="https://teztnets.xyz/mumbainet-about">this page</a> for further details.</p>
<p>We acknowledge that this is coming at a late stage in the current
voting process, and we ask for the community’s understanding and
cooperation in addressing the situation. A full write-up on the
vulnerability will be published when it is safe to do so.</p>
<p>We are confident that the measures outlined in this post will
sufficiently address the present issue and look forward to unleashing
the potential of the <a href="https://research-development.nomadic-labs.com/mumbai-preview.html">ground-breaking
features</a>
contained in Mumbai!</p>Introducing new Octez node logs for better UX2023-03-02T12:00:00+01:002023-03-02T12:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2023-03-02:/introducing-new-octez-node-logs-for-better-ux.html<p>One single line with essential information, and a detailed log kept on disk.</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: Future versions of the Octez node will only output essential information, while a more detailed log is written to disk in the background</strong></p>
<p>Core developers and users of the Octez software suite know it well: the Octez node is pretty verbose. </p>
<p>For every block, the node yells out a whole series of steps taken:</p>
<div class="highlight"><pre><span></span><code>Jan 6 14:33:10.879 - validator.block: prechecked block BLkhtA81uak1SxxCDCVMXEnXCKsJcFLajVT84ZjrFKbxAaJUL4E
Jan 6 14:33:10.900 - validator.block: block BLkhtA81uak1SxxCDCVMXEnXCKsJcFLajVT84ZjrFKbxAaJUL4E validated
Jan 6 14:33:10.900 - validator.block: Request pushed on 2023-01-06T13:33:10.853-00:00, treated in 153us, completed in 45.531ms
Jan 6 14:33:10.920 - prevalidator: switching to new head BLkhtA81uak1SxxCDCVMXEnXCKsJcFLajVT84ZjrFKbxAaJUL4E
Jan 6 14:33:10.920 - prevalidator: Request pushed on 2023-01-06T13:33:10.900-00:00, treated in 308us, completed in 19.333ms
Jan 6 14:33:10.920 - validator.chain: Update current head to BLkhtA81uak1SxxCDCVMXEnXCKsJcFLajVT84ZjrFKbxAaJUL4E
Jan 6 14:33:10.920 - validator.chain: (level 462824, timestamp 2023-01-06T13:33:10-00:00, fitness
Jan 6 14:33:10.920 - validator.chain: 02::00070fe8::::ffffffff::00000000), same branch
</code></pre></div>
<p>To improve the user experience, we’ve decided to make a significant change to these logs. In the next major version of Octez (17.0), the node will output <strong><span class="caps">ONE</span> <span class="caps">SINGLE</span> <span class="caps">LINE</span> for each new block</strong>, plus for injected operations and baked blocks (if running a baker). But don’t worry, all the nitty gritty technical information remains available. </p>
<p>Read on to learn what we did and why. </p>
<h2 id="what-a-node-log-should-do"><a class="toclink" href="#what-a-node-log-should-do">What a node log should do</a></h2>
<p>Logs are a way to communicate information on the state of some software. In the context of the Octez node, the goal is to keep the user aware of the interaction between local Octez software and the Tezos blockchain.</p>
<p>Some examples of what could be going on: </p>
<ul>
<li>The chain is running, node is synced</li>
<li>The node is stuck or bootstrapping</li>
<li>New blocks are being discovered</li>
<li>Connecting or disconnecting to/from peers</li>
<li>Operations are being received</li>
<li>Various errors</li>
<li>…</li>
</ul>
<p>When everything goes well, a new head appears regularly with every new block. Or the node reports receiving new operations. During these peaceful times there is really no need to be verbose. </p>
<p>Still, at every new block, the node is yelling the <em>eight</em> lines in the introduction of this post. These lines contain what to most users must seem like dark magic incantations, like <code>prevalidator</code> and <code>Request pushed</code> or timing in <em>micro-seconds</em>. </p>
<p>Is that interesting for a user? We don’t believe it is. The only pertinent information for a user is :</p>
<div class="highlight"><pre><span></span><code>Hey, it's going well, the level is now 462824 and the most recent head
has the hash BLkhtA81uak1SxxCDCVMXEnXCKsJcFLajVT84ZjrFKbxAaJUL4E.
</code></pre></div>
<h2 id="wait-i-want-the-juicy-details"><a class="toclink" href="#wait-i-want-the-juicy-details">Wait, I want the juicy details!</a></h2>
<p>On the other hand, when issues arise, we want users to be able to communicate errors to relevant people, such as the developer team and incident managers. </p>
<p>In fact, it’s not only the errors we need, but also the state of the node during the last few hours or days. This information can sometimes be even more important than the error itself.</p>
<p>To make the logs less of a pain to read for a user, we need to remove information. But if we want logs to stay useful enough for a developer to fix issues after an incident, we need to keep this information somewhere, and even add more!</p>
<p>So, we’ve split logs into two channels:</p>
<ul>
<li><strong>User logs</strong>: Simple and clean, except for exceptional behaviors or errors. It is shown as the standard output.</li>
<li><strong>Internal logs:</strong> Verbose, but silently written and stored to disk for a limited number of days to avoid excessive use of disk space.</li>
</ul>
<p>For <em>internal logs</em>, we added a new node configuration option (as specified <a href="https://tezos.gitlab.io/user/logging.html">in the documentation</a>), <code>daily-logs</code>, to create a log file every day, which is kept during a user-defined number of days. </p>
<p>Users can find these log files at the path <code><node-data-dir>/daily-logs/</code>. By default, the daily log files will be kept for seven days, in which case the directory shouldn’t exceed <span class="caps">500MB</span>. </p>
<h2 id="stripping-down-the-node-output"><a class="toclink" href="#stripping-down-the-node-output">Stripping down the node output</a></h2>
<p>Now that the important debugging logs are stored elsewhere, we can safely remove information from the <em>user logs</em>. At each block, we now have only one line:</p>
<div class="highlight"><pre><span></span><code>Jan 6 14:33:10.879: head is now BLkhtA81uak1SxxCDCVMXEnXCKsJcFLajVT84ZjrFKbxAaJUL4E (462824)
</code></pre></div>
<p>In short, we</p>
<ul>
<li>Removed the part of the log (like <code>prevalidator</code>, <code>validator.chain</code>) that references a location in the code</li>
<li>Removed a few redundant messages saying the same thing: a new block has been processed and added to the chain</li>
<li>Split some messages containing too much information, like <code>timestamp</code> or <code>fitness</code> that can be found elsewhere anyway.</li>
</ul>
<p>Most of <a href="https://gitlab.com/tezos/tezos/-/milestones/157#tab-issues">this work</a> is expected to appear in Octez suite v17.</p>
<h2 id="whats-next"><a class="toclink" href="#whats-next">What’s next?</a></h2>
<p>The <a href="https://gitlab.com/tezos/tezos/-/milestones/191#tab-issues">next step</a> is mostly a refactoring of the node logging engine, and hopefully removing a deprecated library, <code>lwt_log</code>. This work will allow us to have <em>colors</em> in the logs to emphasize specific information, like when your baker baked a block, or when an error is shown.</p>Announcing Tezos’ 13th protocol upgrade proposal, Mumbai2023-01-17T15:00:00+01:002023-01-17T15:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2023-01-17:/mumbai-announcement.html<p>A joint post from Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda <span class="amp">&</span> Functori.</p><p><em>This is a joint post from Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda <span class="amp">&</span> Functori.</em></p>
<p>We’re happy to announce that Mumbai, our next Tezos protocol proposal,
is ready. As usual, Mumbai’s ‘true’ name is its hash, which is
<code>PtMumbaiiFFEGbew1rRjzSPyzRbA51Tm3RVZL5suHPxSZYDhCEc</code>.</p>
<p>The main features of this protocol upgrade proposal are:</p>
<ul>
<li>
<p><a href="https://research-development.nomadic-labs.com/mumbai-preview.html#smart-rollups-activated">Smart rollups are
activated</a>:
Smart rollups are fully operational on Mainnet offering a powerful
scaling solution. Anyone can deploy decentralized WebAssembly
applications with dedicated computational and networking resources.</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/mumbai-preview.html#epoxy-validity-rollups-on-mondaynet-testnet">Epoxy makes its first
appearance</a>:
Validity Rollups (aka <span class="caps">ZK</span>-rollups) arrive on the Mondaynet
testnet. Epoxy allows for instant finality due to <span class="caps">SNARK</span>’s proof-of-validity.</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/mumbai-preview.html#pipelining-is-fully-deployed-giving-us-15-second-block-times">Block time reduced to 15
seconds</a>:
With improved pipelining fully deployed, block propagation times are
significantly reduced – this allows for the minimal block time to be
cut in half to 15 seconds!</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/mumbai-preview.html#other-improvements">Ticket transfers between
accounts</a>:
In Mumbai, tickets can be <a href="https://gitlab.com/tezos/tezos/-/merge_requests/6108">transferred between user
accounts</a>
(aka. implicit accounts) and not just to/from smart contracts and rollups.</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/mumbai-preview.html#other-improvements">RPCs for ticket
balances</a>:
Two new <span class="caps">RPC</span> endpoints were added to improve the visibility of ticket
ownership. <contract-id>/all_ticket_balances returns a complete list
of tickets owned by a given contract. And
<contract-id>/ticket_balance returns the given-contract’s balance of
the ticket with specified ticketer, content type, and content.</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/mumbai-preview.html#other-improvements">New Michelson
operations</a>:
Michelson opcodes <span class="caps">AND</span>, <span class="caps">OR</span>, <span class="caps">XOR</span>, <span class="caps">NOT</span>, <span class="caps">LSL</span> and <span class="caps">LSR</span> were extended to
support <a href="https://gitlab.com/tezos/tezos/-/merge_requests/6055">logical operations on
bytes</a>,
similar to those on nat. And an opcode was added to <a href="https://gitlab.com/tezos/tezos/-/merge_requests/6681">convert between
bytes and
nat</a>.</p>
</li>
</ul>
<p>To learn more about Mumbai’s contents, see our <a href="https://research-development.nomadic-labs.com/mumbai-preview.html">full preview
post</a>.</p>
<p>In addition to these exciting new features and steady progress, the
Mumbai proposal disables <a href="https://research-development.nomadic-labs.com/toru-introduction.html">Transaction Optimistic
Rollups</a>
(TORUs) on <code>Mainnet</code>. TORUs were the first optimistic rollup
implementation on <code>Mainnet</code> enabled with
<a href="https://research-development.nomadic-labs.com/jakarta-is-live.html">Jakarta</a>
mid 2022, and the first step of the scalability <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html">roadmap
presented</a>
last March.</p>
<p>Transaction rollups were always intended to be an temporary solution,
as clearly indicated by the sunset they were released with. Now that
smart rollups are reaching Tezos <code>Mainnet</code>, it makes little sense to
keep transaction rollups enabled, as this functionality can easily be
implemented through Smart Rollups. It is important to understand that,
if Mumbai is voted in by the community, the transaction rollup subsystem
will be completely disabled as soon as the protocol becomes activate on
<code>Mainnet</code>. This means tickets deposited on transaction rollups will be
lost forever, and operators will not be able to reclaim their ꜩ 10,000
bond. Currently, no transaction rollups have been originated on <code>Mainnet</code>.</p>
<p>The <a href="https://tezos.gitlab.io/protocols/016_mumbai.html">changelog</a>
provides a detailed list of changes, and a general technical overview
of Mumbai can be found in the protocol proposal’s <a href="https://tezos.gitlab.io/mumbai/protocol.html">technical
documentation</a>.</p>
<p>Note that, if Mumbai is voted in by the community, upgrading to Octez
v16.0 (or later) will be necessary for participating in consensus. A
release candidate for Octez v16.0 will be published in the coming
days, and a dedicated protocol test network <code>Mumbainet</code> is also
scheduled to begin soon. More information about the test network will
be available on <a href="https://teztnets.xyz/">https://teztnets.xyz/</a>.</p>
<h2 id="smart-rollups-and-epoxy-testers-wanted"><a class="toclink" href="#smart-rollups-and-epoxy-testers-wanted">Smart Rollups and Epoxy testers wanted</a></h2>
<p>First, Smart Rollups will be active on <code>Mumbainet</code> soon. Functioning
rollups implemented in Rust are currently running on <code>Mondaynet</code>. We
highly encourage ecosystem participants to experiment and build with
these rollups. In roughly 2 months they are expected to activate on
<code>Mainnet</code>, assuming Mumbai is voted in. Smart Rollups are instrumental
in our efforts to reach <a href="https://research-development.nomadic-labs.com/smart-rollups-are-coming.html">1 millon
<span class="caps">TPS</span></a>.</p>
<p>Second, Epoxy - Tezos’ validity rollup (aka <span class="caps">ZK</span>-rollup) solution makes
its way onto <code>Mondaynet</code>. It’s not, however, part of the Mumbai
protocol upgrade, as more time needs to be spent on validation,
testing, and integration with ecosystem tools before they can be
activated on Mainnet. We ask ecosystem participants to start
experimenting with Epoxy on <code>Mondaynet</code>.</p>
<p>Broad testing and feedback from the ecosystem is invaluable in our
efforts to minimize the risk of undetected issues upon <code>Mainnet</code>
activation.</p>
<p>Read more about Tezos testnets <a href="https://teztnets.xyz/">here</a>, and
don’t hesitate to reach out in the <a href="https://tezos-dev.slack.com/">Tezos Developer
Slack</a> or in the <a href="https://discord.com/invite/5v5QVKjvc2">Tezos
Discord</a> if you need help
getting started.</p>
<p>Anyone interested in getting started with building a smart rollup node
can reach out to
<a href="mailto:contact@nomadic-labs.com">contact@nomadic-labs.com</a>.</p>Incident Report: slow consensus on block #3,019,8512023-01-05T18:00:00+01:002023-01-05T18:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2023-01-05:/incident-report-slow-consensus-on-block-3019851.html<p>A low-probability scenario caused a temporary slowdown of the network. This is a known possibility and intended behavior of the Tenderbake consensus algorithm.</p><!-- Imported from
https://docs.google.com/document/d/1F6jUdUOZLvOQvcJ6SLFurRDKfPIg3CdDplV7vjx9C4E/edit#heading=h.idu41eirf3a5
-->
<p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: A low-probability scenario caused a temporary slowdown of the
network. This is a known possibility and intended behavior of the
Tenderbake consensus algorithm.</strong></p>
<p>On January 1st, around 14:13 <span class="caps">UTC</span>, the Tezos Mainnet experienced a
major slowdown: ~52 minutes elapsed between block levels
<a href="https://tzkt.io/3019850">#3,019,850</a> and
<a href="https://tzkt.io/3019852">#3,019,852</a>, due to block
<a href="https://tzkt.io/3019851">#3,019,851</a> requiring 18 rounds to reach
consensus. Network endorsing power recovered immediately on the
following block, producing a slower round 0 block. After
<a href="https://tzkt.io/3019853">#3,019,853</a>, block time stabilized to the
expected 30 second time between blocks.</p>
<p>Following the incident, we’ve examined how this slowdown came to
happen, and why the network failed to reach a quorum on earlier rounds.</p>
<p>Today, we are confident that no bug in either the Tenderbake consensus
algorithm or its implementation by the Octez baker was behind this
incident. The results of our investigation point towards a
particularly unlikely, yet known, scenario.</p>
<p>Below, we will detail how it unfolded. But first, a short summary:</p>
<ul>
<li>
<p><strong>Block production did not stop, Tenderbake kept running:</strong> it took
43 minutes for the network to agree on the next blockchain head, as
a result of <a href="https://tzkt.io/3019851">#3,019,851</a> being finalized at
round 17. But, proposals were made in all previous rounds at that
level. Remember that Tenderbake entails a trade off in choosing
<a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html#nakamoto-is-live-classical-bft-is-safe">network safety over network
liveness</a>.</p>
</li>
<li>
<p><strong>Network was perfectly safe:</strong> the behavior observed fits within a
known slow consensus scenario for Tenderbake: the endorsing stake
was split roughly even between bakers which had locked their
endorsement for the Round 0 block and those who did not, without
reaching the 67% endorsing power requirement.</p>
</li>
<li>
<p><strong>Liveness was slow, but kicking</strong>: it took 17 extra rounds for one
of the locked bakers to re-propose the locked payload. It
immediately got sufficient endorsements for the chain to move on.</p>
</li>
<li>
<p><strong>A very late block proposal was the trigger:</strong> the scenario was
triggered by a block proposal arriving unusually late within the
expected time slot. The reasons for this delay are still under
investigation. However, it was less pressing than assessing whether
the Octez baker implemented Tenderbake correctly.</p>
</li>
<li>
<p><strong>An unlikely yet known (and tested!) scenario</strong>: the particulars of
this slow consensus scenario (detailed below) make it a quite
unfortunate incident. Moreover, similar scenario were already
thoroughly tested both in the Octez baker’s test suite and using
Functori’s
<a href="https://functori.com/blog/blog-playing-consensus-scenarios-on-Tenderbake-with-Mitten.html">Mitten</a>
simulation toolkit.</p>
</li>
<li>
<p><strong>Not related to baker crashes:</strong> we had first observed reports of
baking nodes crashing due to a lack of available storage space. This
working hypothesis, mentioned in our early messages on Sunday,
turned out to be a false lead. It is not, a priori, connected to the
slow consensus scenario that unfolded.</p>
</li>
</ul>
<p>So, what happened?</p>
<h2 id="in-focus-the-consensus-committee-for-3019851"><a class="toclink" href="#in-focus-the-consensus-committee-for-3019851">In focus: the consensus committee for #3,019,851</a></h2>
<p>To understand what happened, we dug into the committee for #3,019,851,
and analyzed its interactions before it reached a quorum. We relied on
<a href="https://gitlab.com/nomadic-labs/teztale">Teztale</a>, a live
consensus-introspection tool, currently being developed at Nomadic
Labs. We started building Teztale to inform our Incident Response Team
in situations like this.</p>
<p><strong>As an aside,</strong> the description that follows is fairly technical and
requires some understanding of how Tenderbake works. If you are not
familiar with the concepts or the terminology, we highlight a few resources:</p>
<ul>
<li>
<p>The Lima protocol’s <a href="https://tezos.gitlab.io/active/consensus.html">Consensus documentation
entry</a>.</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">A look ahead to
Tenderbake</a>
(blog post).</p>
</li>
<li>
<p><a href="https://functori.com/blog/blog-tenderbake-baker-as-state-machine.html">Tenderbake’s Baker as a
StateMachine</a>
(blog post).</p>
</li>
<li>
<p><a href="https://arxiv.org/abs/2001.11965">A Solution to Dynamic Repeated Consensus for
Blockchains</a> (research article).</p>
</li>
</ul>
<h3 id="round-0"><a class="toclink" href="#round-0">Round 0</a></h3>
<figure
align="center"
id="Figure-1">
<img src="/images/slow-consensus-Round0.png"
alt="Reception times for #3,019,851 at Round 17"
width="90%">
</img>
</figure>
<p>Let’s plunge into the consensus committee for #3,019,851. The histogram
above tracks the reception of the block proposal at round 0, and its
associated preendorsements and endorsements. Here, the X axis tracks
the elapsed time since round rights were enabled, and the Y axis plots
the number of delegates seen (pre)endorsing this proposal.</p>
<p>The rights to propose a block at round 0 at level #3,019,851 were
enabled at <code>2023-01-01T14:14:29Z</code> – that corresponds to the origin of
the plot. The baker holding those rights proposed a payload, whose
block hash was <code>BLZL7AtZKP21eQKEESymTCJYvfV5fpuh6ZZQXffTwg4QstzfxoD</code>.</p>
<p>However, this block proposal was not seen (and validated!) by our node
running Teztale, or any of the nodes we have access to, until
<code>14:14:54Z</code> – denoted by the red vertical bar in the histogram. That
is, the block payload could only be (pre)endorsed 25 seconds after
rights were enabled. This is awkward, as most bakers usually produce
and propagate blocks significantly faster than this.</p>
<p>Given a 30 seconds round duration, Round 0 was scheduled to finish at
<code>14:14:59Z</code>. This left only 6 seconds for the network to reach
consensus on this block. It was clearly not the case: preendorsements
and endorsements for this round arrived too late to our node, as the
histogram shows.</p>
<p>Let’s focus on the (pre)endorsements in this round. In this particular
committee, 229 bakers (out of the 240 that had endorsing rights)
managed to inject a preendorsement within this period. Then, 122
bakers (accounting for 43.83% of the endorsing power) observed in time
<a href="https://tezos.gitlab.io/active/consensus.html#quorum-active">that a pre-quorum was
reached</a>,
and proceeded to endorse that block – witnessed by the violet bars in
the histogram.</p>
<p>This action “locked” them to the endorsed payload content for all
successor rounds at level #3,019,851. This means they would only endorse
<a href="https://tezos.gitlab.io/alpha/consensus.html#overview">re-proposals of the original block
payload</a>.</p>
<p>Given that 43,8% of endorsing power is obviously more than 33% but far
from the required 67%, the rules of Tenderbake specify that the only
way to move forward in this scenario is when one of the “locked”
bakers re-proposes this round proposal’s payload content.</p>
<p>That is, when it is their turn to do so.</p>
<h3 id="rounds-1-16"><a class="toclink" href="#rounds-1-16">Rounds 1 – 16</a></h3>
<p>Here is when Fate kicks in. Unfortunately, all bakers with proposing
rights between Rounds 1 and 16 were part of the 104 bakers (amounting
to the 56.17% of the stake) that had not locked to the payload of
round 0!</p>
<p>The probability of arriving at this scenario, given the stake
distribution in play, was 0.5617^16 = 0.0098191%. That is, in roughly
1 out of 10000 possible committee configurations, the right to propose
a block in the next 16 rounds would be given to a baker which had not
locked its commitment to the payload proposed at Round 0. And this was
the case.</p>
<p>Indeed, in every round between Round 1 and Round 16, the designated
delegate proposed a block. Yet, neither of them were locked, so they
proposed a fresh proposal with a new payload instead.</p>
<p>They were all doomed to fail: their proposals would never be endorsed
by the locked delegates and it was not possible to reach the minimal
endorsement requirements without them.</p>
<p>The network was both safe and live. Yet, the dice had rolled a bad
hand, and there was regrettably nothing to do but wait for fairer winds.</p>
<h3 id="round-17-and-beyond"><a class="toclink" href="#round-17-and-beyond">Round 17 and beyond</a></h3>
<figure
align="center"
id="Figure-2">
<img src="/images/slow-consensus-Round17.png"
alt="Reception times for #3,019,851 at Round 17"
width="90%">
</figure>
<p>Finally, it was the turn for
<a href="https://tzkt.io/tz1gUNyn3hmnEWqkusWPzxRaon1cs7ndWh7h/operations">tz1gUNyn3hmnEWqkusWPzxRaon1cs7ndWh7h</a>
to propose a block at Round 17. This delegate was one of the delegates
locked on the payload proposed at Round 0, and correctly re-proposed
it. The resulting block proposal,
<code>BLfpYc4HvRpyacP6f75hpTgRJjzzpfyQRaxzNQys5gByEdr6T24</code>, was produced
with timestamp <code>2023-01-01T14:56:59Z</code>, as expected.</p>
<p>Most delegates in the committee endorsed the block proposal in time,
as seen in the histogram above, resulting in 6,982 out 7,000 endorsed
slots. Having reached consensus on the head of the chain, it became
<a href="https://tzkt.io/3019851">#3,019,851</a> and the network finally moved on.</p>
<p>Note that the next block <a href="https://tzkt.io/3019852">#3,019,852</a> was
produced at Round 0. Still, the <a href="https://tezos.gitlab.io/active/consensus.html#time-between-blocks">“time between
blocks”</a>
#3,019,852 and #3,019,851 is around 5 minutes because the timestamp
of the former corresponds to the end of the last round of the latter,
namely the end of Round 17. Its successor, another Round 0 block,
stabilized time between blocks to 30 seconds.</p>
<h2 id="lessons-and-take-aways"><a class="toclink" href="#lessons-and-take-aways">Lessons and take-aways</a></h2>
<p><strong>The foremost conclusion is that what we experienced is just
Tenderbake at work.</strong></p>
<p>We are relieved to see the implementation of Tenderbake working as
expected in a slow consensus scenario. Still, we regret having
experienced the worst slowdown in block propagation since <a href="https://research-development.nomadic-labs.com/activating-tenderbake.html">the
adoption of Tenderbake on Tezos
Mainnet</a>:</p>
<ul>
<li>
<p><a href="https://tzkt.io/2244609">#2244609</a>, the first block of the
Tenderbake era required 15 rounds to reach consensus, as some
delegates needed time to adapt to the migration.</p>
</li>
<li>
<p><a href="https://tzkt.io/2244610">#2244610</a>, the second block in the
Tenderbake era, required 13 rounds.</p>
</li>
<li>
<p><a href="https://tzkt.io/2490369/operations">#2490369</a>, the first block of
<a href="https://research-development.nomadic-labs.com/jakarta-is-live.html">Jakarta</a>,
required 12 rounds – due to an expected longer migration due to
patching contracts during stitching.</p>
</li>
</ul>
<p>Looking ahead, we consider possible optimizations targeting scenarios
like the one described above. For instance, Tenderbake could be less
aggressive when increasing round duration. Another possibility is to
allow bakers which are <strong>not locked</strong> to still consider late
preendorsements for older payloads, instead of just discarding
them. This would allow them to re-propose a block consistent with the
potentially locked payload, leading to faster agreement.</p>
<p>When we first implemented Tenderbake, we gave more priority to the
safety of the implementation over heuristics to improve
liveness. Today, as we propose to further reduce <a href="https://research-development.nomadic-labs.com/mumbai-preview.html">the time between
blocks</a>,
these optimizations become more relevant, and we look to implement
them in the near future.</p>Lima, the latest Tezos upgrade, is LIVE!2022-12-19T03:30:00+01:002022-12-19T03:30:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2022-12-19:/lima-is-live.html<p>Tezos has activated its twelfth protocol upgrade, Lima.</p><p>On December 19 2022 01:57:59 <span class="caps">UTC</span>, the Tezos blockchain successfully
upgraded by activating the Lima proposal at block
<a href="https://tzkt.io/2981889">#2,981,889</a>.</p>
<p>This 12th Tezos protocol upgrade was jointly developed by Nomadic
Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda, <span class="amp">&</span> Functori.</p>
<p>Lima introduces:</p>
<ul>
<li>
<p><a href="https://research-development.nomadic-labs.com/lima-preview.html#more-pipelining-for-a-faster-blockchain">More
pipelining</a>:
Speeding up propagation of operations and blocks to enable higher
Layer 1 throughput on Tezos. This work is also the foundation for
reducing block time to 15 seconds in <a href="https://research-development.nomadic-labs.com/mumbai-preview.html">the upcoming Mumbai
proposal</a>.</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/lima-preview.html#this-ones-for-the-bakers-introducing-consensus-keys">Consensus
keys</a>:
This much requested feature lets bakers change their key for signing
blocks and consensus operations without changing the baker’s public address.</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/lima-preview.html#improvements-to-tickets">Improvements to
Tickets</a>:
Ticket ownership updates are now part of transaction receipts, which
helps indexers keep track of tickets. Also, zero-amount tickets will
be deprecated.</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/lima-preview.html#ghostnet-fixes">Ghostnet
fixes</a>:
Lima fixes two problems that arose during the migration from Jakarta
to Kathmandu on the permanent ‘Ghostnet’ test network.</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/lima-preview.html#other-changes">Liquidity Baking sunset
removed</a>:
The subsidy can be turned off with the moving-average toggle
introduced with the Jakarta upgrade.</p>
</li>
<li>
<p><a href="https://research-development.nomadic-labs.com/lima-preview.html#other-changes">Temporary Timelock
deprecation</a>:
Due to a discovered vulnerability, origination of new contracts
using Timelock are disabled while a safer mechanism is developed.</p>
</li>
</ul>
<p>To learn more about Lima’s contents, see our<a href="https://research-development.nomadic-labs.com/lima-preview.html"> preview
post</a>.</p>
<p>The <a href="https://tezos.gitlab.io/protocols/015_lima.html">changelog</a>
provides a detailed list of changes, and a general technical overview
of Lima can be found in the protocol proposal’s <a href="https://tezos.gitlab.io/lima/protocol.html">technical
documentation</a>.</p>
<h2 id="octez-v151"><a class="toclink" href="#octez-v151">Octez v15.1</a></h2>
<p>With Lima activated, we highly recommend upgrading to Octez v15.1, as
this version fixes a bug in the bootstrap pipeline that would make
v15.0 nodes apply blocks without prechecking them first.</p>
<p>As always, we encourage the ecosystem to participate in testnets. Read
more about Tezos testnets <a href="https://teztnets.xyz/">here</a>, and don’t
hesitate to reach out in the <a href="https://tezos-dev.slack.com">Tezos Developer
Slack</a>
or in the <a href="https://discord.com/invite/5v5QVKjvc2">Tezos Discord</a> if
you need help getting started.</p>Pruning the context — and other seasonal activities2022-12-12T18:00:00+01:002022-12-12T18:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2022-12-12:/pruning-the-context-and-other-seasonal-activities.html<p>A long awaited feature is now available with the lastet major Octez release: context pruning, aka garbage collection of the context. In this article we explore why this was needed, and how it impacts daily life of node operators.</p><p><em><span class="caps">TL</span>;<span class="caps">DR</span> The latest Octez major release introduced a long-awaited
feature: context pruning for rolling and full nodes’ storage — aka
the garbage collection of the context storage. Context pruning
significantly reduces disk usage of Octez nodes, lowering the barrier
of entry, and improving the quality of life of node operators and bakers.</em></p>
<!---
https://gitlab.com/nomadic-labs/research-development.nomadic-labs.com/-/issues/54
-->
<!--- Spell-checked with an en_US dictionary -->
<!--
Story:
- chains grows every day
- each blocks are stored on disk in the "chain's history" (aka store)
- each balance updates and smart contracts are stored on disk in the
"ledger state" (aka context)
- this requires a lot of space
- useful for "archaeology" (role of indexers and archive nodes)
- not necessary for block validation/consensus/reward distribution
--->
<h2 id="where-does-all-this-data-come-from"><a class="toclink" href="#where-does-all-this-data-come-from">Where does all this data come from?</a></h2>
<p>A desirable property of any blockchain is that the chain <em>grows</em>. Most
blockchains share the notion of a “genesis block” — a Big Bang event
where the first block was produced, and information is added
continuously and (hopefully) forever: transactions, smart contract
calls, <span class="caps">NFT</span> mints, etc. Ultimately, this ever-growing amount of data
has to be stored by node operators.</p>
<!-- From that moment on these (countably) infinite perpetual
machines, -->
<p>In Tezos parlance, we speak of <em>context</em> when referring to the ledger
state. For each blockchain level, the context reflects the account
balances, the code and storage of smart contracts and other global
data pertinent to the business logic of the <a href="https://tezos.gitlab.io/active/protocol_overview.html">Tezos economic
protocol</a>:
information about
<a href="https://tezos.gitlab.io/active/proof_of_stake.html">stake</a> and
<a href="https://tezos.gitlab.io/active/consensus.html">consensus rights</a>,
<a href="https://tezos.gitlab.io/active/michelson.html#operations-on-tickets">tickets</a>,
<a href="https://tezos.gitlab.io/active/global_constants.html">global
constants</a>, and
more, that are updated by the block operations of a given level. </p>
<p>Moreover, the context storage in the Octez node is <em>versioned</em>: you
can think of the context store as a <em>key-value store</em> mapping each
block level (or each block hash) to the full context at that level. Of
course, the real implementation is smarter (and more efficient). The
Octez node relies on the <a href="https://irmin.org/">Irmin</a> storage system,
developed and maintained by <a href="https://tarides.com">Tarides</a>, which
compresses and optimizes this history. Irmin works in a similar way
that <a href="https://en.wikipedia.org/wiki/Git">Git</a> works, storing <code>diffs</code>
between different versions of the same entry rather than recording
each version of an entry in full<sup id="fnref:simple"><a class="footnote-ref" href="#fn:simple">1</a></sup>. Still, this takes <em>some</em> space.</p>
<p>Thus, starting from <a href="https://tzstats.com/0">its origin block</a>, almost
3 million blocks ago<sup id="fnref:time"><a class="footnote-ref" href="#fn:time">2</a></sup> the size of the Tezos storage never
stopped growing.</p>
<!-- **Why do we need to keep all this data around?** The answer is simple: -->
<!-- _verifiability_. The consistency of the Tezos (or any blockchain) -->
<!— ledger depends on being able to replay — that is, to reproduce — the —>
<!-- complete history of the Tezos (or any!) blockchain from its genesis -->
<!-- block to a desired point in time. It is paramount then to be able to -->
<!-- find (if necessary or if we want to), the information from any given -->
<!-- block, at all times. Then, storing the results of those computations -->
<!-- (the context is computed and updated each time a node validates a new -->
<!— block) makes sense for efficiency purposes as well — compute once, —>
<!-- read from disk later, as those values are immutable[^ohdear]. -->
<p><strong>Why do we need to keep all this data around?</strong> There are two parts
to the answer to this question. First, it is paramount enough
participants do: For the sake of <em>verifiability</em>, we need to be able
to replay — that is, to reproduce — the complete history of the
Tezos blockchain from its genesis block to a desired point in time and
see the resulting context is the expected one. This tasks entail being
able to find the contents from any given block, at all times so <a href="https://tezos.gitlab.io/active/validation.html#validation-and-application">we
can apply
them</a>
on top of the predecessor block’s context.</p>
<p>Then, storing the results of those computations (the context is
computed and updated each time a node validates a new block) makes
sense for efficiency purposes as well — compute once, read from disk
later, as those values are immutable<sup id="fnref:ohdear"><a class="footnote-ref" href="#fn:ohdear">3</a></sup>. For instance, a baker
should not need to replay the chain each time to find out, say, what
were its delegators balances during the recent cycles in order to pay
delegation rewards.</p>
<p><strong>Do we all need to keep this data around?</strong> Indeed, not every node
operator requires access to the complete history of the chain: a baker
doesn’t need to know all the chain’s history to bake new blocks; and
we rather rely on indexers and block explorers to query operation
details from any given block in the past.</p>
<p>To cater these different needs, the Octez implementation provides
different storage modes, known as <a href="https://tezos.gitlab.io/user/history_modes.html">history
modes</a>: <em>rolling</em>,
<em>full</em>, and <em>archive</em>. Nodes running in archive history mode keep all
data (and metadata) from all operations since genesis, while nodes
running rolling or full history modes keep less information, and hence
demand a leaner footprint. Indexers and block explorers rely on
<em>archive</em> nodes, whereas bakers are
<a href="https://opentezos.com/baking/cli-baker/#lets-dive-into-the-practical-part">recommended</a>
to rely on <em>full</em> nodes.</p>
<p>Even for users running the lighter <em>rolling</em> and <em>full</em> history modes,
keeping multiple versions of the context imposes a significant disk
space overhead.</p>
<!-- GAD maybe we want to say here what was the blocker/reason not to
implement this before, if it wasn't just that Irmin didn't work
well-->
<p>Starting with <a href="https://tezos.gitlab.io/releases/version-15.html#version-15-0">Octez
v15.0</a>,
we incorporate the first version of a long-awaited feature, dubbed
<em>context pruning</em><sup id="fnref:pruning"><a class="footnote-ref" href="#fn:pruning">4</a></sup>. This recurrent maintenance process
drastically reduces the disk footprint of the context store for <em>full</em>
and <em>rolling</em> nodes, and the overall free space required to operate an
Octez Tezos node. Moreover, users should notice an overall improvement
in node performance as well!</p>
<p>This feature is a result of our ongoing collaboration with
<a href="https://tarides.com/">Tarides</a>. The bulk of the work entailed working
under the hood in Irmin, to implement the necessary changes in the
overall architecture so that the Octez node could profit from this new functionality.</p>
<p>Tarides has published a quite detailed, in-depth
<a href="https://tarides.com/blog/2022-11-10-towards-minimal-disk-usage-for-tezos-bakers">article</a>
about the technical aspects of context pruning, <a href="https://en.wikipedia.org/wiki/Garbage_collection_(computer_science)">garbage
collection</a>,
and the intricacies of its implementation within Irmin.</p>
<p>Today, we rather focus on a higher level description of this feature
and how it fits in with the overall Octez architecture, and provide a
report of our early experimentation on the impact of the new feature
on our working Mainnet nodes.</p>
<h2 id="whats-context-pruning-then"><a class="toclink" href="#whats-context-pruning-then">What’s context pruning then?</a></h2>
<!-- What is GC:
- Context Garbage Collection (or Context Pruning ?)
- full and rolling (especially) nodes aims to keep only the recent
data of the chain to lower the disk footprint of the node's data
directory
- blocks and their associated ledger state can be removed on regular
basis
- this requires a maintenance procedure to filter old/obsolete/useless
data
- can be configured (see doc of history modes)
-->
<p>As we mentioned earlier, there are three different history modes in
Tezos. And here today we focus on two: <em>rolling</em> and <em>full</em>. These two
history modes aim to keep only relatively recent chain data, and
reduce the disk footprint of the node’s data directory. The full
history mode stores the minimal data necessary to replay the chain
from genesis, but doesn’t snapshot context information from older
levels, like older balances or staking rights. The rolling mode is
even more aggressive, as it keeps information from just a limited
number of cycles — on Mainnet, the default is <a href="https://tezos.gitlab.io/shell/storage.html#store">the last
6</a>.</p>
<p>Still, the way in which <code>irmin-pack</code> (the Irmin back-end used by and
conceived for Tezos) was designed did not allow to fully reap the
rewards of running leaner history modes: it was not possible to
“delete” the information from dead, unreachable objects (the data from
blocks deemed to be too old to be kept in the
<a href="https://research-development.nomadic-labs.com/introducing-a-new-storage-backend-for-octez.html">store</a>),
and reclaim valuable free space without prohibitive performance trade-offs.</p>
<p>The new Irmin version <a href="https://tarides.com/blog/2022-11-10-towards-minimal-disk-usage-for-tezos-bakers#improve-space-usage-with-context-pruning">addresses this
limitation</a>
by re-structuring the design of the context store.</p>
<p><a id="gc"><strong>How does context pruning work?</strong></a> Context pruning is,
in essence, an automated garbage collector (<span class="caps">GC</span>) mechanism. It executes
periodically, and asynchronously, with regard to other Irmin
tasks. The <span class="caps">GC</span> determines a window of data to be kept, and iterates
over the data on disk to extract the information from blocks belonging
to that window which needs to be preserved — those from <em>reachable</em>
objects only. When all the reachable data is extracted, the storage
will switch to that newly garbage collected data, and drop the former
one, which contains <em>unreachable</em> objects.</p>
<p>As a result, once context pruning kicks in, the disk footprint from
the context increases linearly in its true size instead of the age of
the data directory, eliminating the need for manual maintenance
operations on long-running Octez nodes. In addition, the reduced
footprint entails a better performing Octez node, as read and write
accesses become cheaper.</p>
<p>We refer to <a href="https://tarides.com/blog/2022-11-10-towards-minimal-disk-usage-for-tezos-bakers">Tarides’ blog
post</a>
for further details on how the process works. In the following
section, we revisit the impact the new context storage has on Mainnet nodes.</p>
<!--
- defines a window of data to keep
- iters over the data on disk and extract everything we want to keep
- requires to "double the context's space to do so" (to be optimized
in a future release)
- disk footprint is not growing steadily
- disk footprint is almost constant (not really as there are always
new accounts/contracts added to the chain) new accounts/contracts
added to the chain)
- See full details and specifications on Tarides' blog post
-->
<h2 id="putting-pruning-in-context"><a class="toclink" href="#putting-pruning-in-context">Putting pruning in context</a></h2>
<p>As excited as you are to test this new feature, we ran a limited
experiment to assess the impact in our Mainnet nodes.</p>
<p>The experiment consisted in importing the same rolling <code>Mainnet</code>
snapshot on two similar machines:</p>
<ul>
<li>one running a vanilla Octez v15.0 node — that is, with context
pruning enabled; and,</li>
<li>one machine running a v15.0 Octez node with context pruning manually
(and artificially) disabled — so, no pruning for you.</li>
</ul>
<p>The purpose of the experiment was to simulate running both nodes under
the same conditions and throughout a period of time (validating around
40 cycles in a row) where we could witness the effect of several calls
to the context pruning mechanism.</p>
<p><a href="##Figure-1">Figure 1</a> below plots the disk footprint of context
storage on both machines; the raw data snapshot can be explored
interactively
<a href="https://snapshots.raintank.io/dashboard/snapshot/cbAD4D6pxPBLxBzok1PEiCJ3QZjmNdF0?orgId=2">here</a>.</p>
<!-- forever add transactions, smart contracts, and
other --><!-- operations. --><figure
align="center"
id="Figure-1">
<a href="https://snapshots.raintank.io/dashboard/snapshot/cbAD4D6pxPBLxBzok1PEiCJ3QZjmNdF0?orgId=2">
<img src="/images/context-pruning-experiment.png"
alt="Contrasting context storage footprints on Octez v15.0 nodes: context pruning enabled vs. context pruning disabled."
width="90%">
</a>
<figcaption align="center">
<b>Figure 1:</b> Contrasting context storage footprints on Octez v15.0
nodes: context pruning <span style="color:green">
<b>enabled</b></span> vs. context pruning <span
style="color:GoldenRod"><b>disabled</b></span>.
</figcaption>
</figure>
<p>We see that the disk footprint from the stored context for the
“artificially unpruned” node quickly rose up to ~140 <span class="caps">GB</span>. On the
vanilla Octez v15.0 node with context pruning enabled, the size of the
context storage on disk is significantly smaller: it oscillates
between ~<span class="caps">30GB</span> and ~60 <span class="caps">GB</span> — the spikes are caused by the duplication
needed before the context pruning calls, and they will be lowered in
upcoming Octez releases. <strong>Still, that’s a 2x — 4x reduction in disk footprint!</strong></p>
<h2 id="how-to-profit-from-context-pruning"><a class="toclink" href="#how-to-profit-from-context-pruning">How to profit from context pruning</a></h2>
<p>To benefit from context pruning, there are a few simple steps to
follow. As the storage requires a particular format introduced in
<a href="https://tezos.gitlab.io/CHANGES.html#id13">Octez v13.0</a> (storage
version <code>0.0.8</code>), it is mandatory to make sure that your node is
running with such a compatible storage format.</p>
<p>The easiest way to make sure of that is to import a fresh snapshot
using an Octez v15.1 node. You will find detailed instructions <a href="https://tezos.gitlab.io/user/snapshots.html#importing-a-snapshot">on how
to import
snapshots</a>
in the online docs.</p>
<p>If the data directory was imported recently enough, that is by a node
running Octez v13.0 or later, then you can skip the step to import the
snapshot. However, the first context pruning operation for nodes that
have been running on the same data directory for a long time can take
a while, and use a significant amount of memory. To avoid this, it is
also recommended to import a fresh snapshot.</p>
<p>As context pruning is enabled by default, there is nothing much to
do. If you have configured your node as usual, you should be all
set. If something is amiss though, a warning message like this will be
printed while starting your node.</p>
<div class="highlight"><pre><span></span><code><span class="o">...</span> <span class="p">:</span> <span class="n">garbage</span> <span class="n">collection</span> <span class="ow">is</span> <span class="ow">not</span> <span class="n">fully</span> <span class="n">enabled</span> <span class="n">on</span> <span class="n">this</span> <span class="n">data</span> <span class="n">directory</span><span class="p">:</span>
<span class="n">context</span> <span class="n">cannot</span> <span class="n">be</span> <span class="n">garbage</span> <span class="n">collected</span><span class="o">.</span> <span class="n">Please</span> <span class="n">read</span> <span class="n">the</span> <span class="n">documentation</span> <span class="ow">or</span>
<span class="kn">import</span> <span class="nn">a</span> <span class="n">snapshot</span> <span class="n">to</span> <span class="n">enable</span> <span class="n">it</span><span class="o">.</span>
</code></pre></div>
<p>That’s pretty much all you should care about. Context pruning being an
internal maintenance procedure, we have not considered (yet) scrapping
specific <a href="https://tezos.gitlab.io/user/node-monitoring.html">telemetry to monitor
it</a> — other than,
of course, witnessing its effect by measuring the full size as in
<a href="##Figure-1">Figure 1</a> above<sup id="fnref:monitor"><a class="footnote-ref" href="#fn:monitor">5</a></sup>. This maintenance procedure is
executed at each cycle dawn. Note that, if you have imported a fresh
snapshot, context pruning will only occur after having validated
blocks from 6 complete cycles, i.e. around 3 weeks<sup id="fnref:consensus"><a class="footnote-ref" href="#fn:consensus">6</a></sup>.</p>
<h2 id="looking-ahead"><a class="toclink" href="#looking-ahead">Looking ahead</a></h2>
<p>While a significant part of our collective focus in 2022 has been on
developing and deploying Tezos (vertical) <a href="https://research-development.nomadic-labs.com/smart-rollups-are-coming.html">scalability
solutions</a>
— like
<a href="https://research-development.nomadic-labs.com/toru-introduction.html">transaction</a>
or <a href="https://research-development.nomadic-labs.com/next-generation-rollups.html">smart
rollups</a>
— that doesn’t mean we have forgotten about making the Tezos Layer 1
<a href="https://research-development.nomadic-labs.com/lima-preview.html#more-pipelining-for-a-faster-blockchain">faster</a>
and <a href="https://research-development.nomadic-labs.com/activating-tenderbake.html">more
secure</a>. Together
with earlier improvements <a href="https://tarides.com/blog/2022-04-26-lightning-fast-with-irmin-tezos-storage-is-6x-faster-with-1000-tps-surpassed">to the Octez node’s storage
layer</a>,
context pruning makes the overall node leaner, demanding less
resources, and improving overall quality of life for node operators
and bakers. In addition to that, it also improves the performance: a
lighter context implies faster read/writes.</p>
<p><strong>Further upgrades to Irmin and context pruning.</strong> As Tarides’ blog
<a href="https://tarides.com/blog/2022-11-10-towards-minimal-disk-usage-for-tezos-bakers#further-optimisations-in-the-octez-storage-layer">post
announces</a>,
this is a first deployment of the context pruning mechanism, and more
improvements to Irmin are already in development for future Octez
releases. These target:</p>
<ul>
<li>Avoiding duplication of the context before the actual pruning takes
place (effectively keeping the footprint around <span class="caps">40GB</span>).</li>
<li>Improving the overall performance of the mechanism to be less
resource intensive.</li>
<li>Adding “a lower” layer to archive nodes, so that old data will be
kept in it. The live data will remain in an lighter “upper layer”,
allowing an overall performance increase.</li>
</ul>
<p><strong>Other changes in Octez v15.x</strong> In addition to the context pruning
feature described in this article, the latest Octez major release
includes other minor changes. It is also worth pointing out it was
recently superceeded by a new minor release, <a href="http://tezos.gitlab.io/releases/version-15.html#version-15-1">Octez
v15.1</a>,
to correct a small-but-critical bug in the <a href="https://tezos.gitlab.io/shell/validation.html#validator">bootstrap
pipeline</a>. We
invite you to take a careful look at the
<a href="https://tezos.gitlab.io/CHANGES.html#version-15-1">Changelog</a> before
upgrading your node. Another <strong>breaking change is a complete renaming
of all executables</strong>: <code>tezos-client</code> is now named <code>octez-client</code>,
<code>tezos-baker-014-PtKathma</code> becomes <code>octez-baker-PtKathma</code>, etc. This
was an overdue change, to reflect that the Octez implementation is one
of possibly many Tezos node implementations, as described when <a href="https://research-development.nomadic-labs.com/announcing-octez.html">we
announced the Octez
name</a>. In
a previous <a href="https://research-development.nomadic-labs.com/filename-renaming-octez-v15.html">blog
post</a>,
we have expanded on this rationale, and discussed the
backward-compatibility mitigation provided in the release.</p>
<!-- TODO: We need to fix to v15.1 if we don't publish before Dec 1st
--->
<p><strong>Getting ready for Lima!</strong> Note that the Lima protocol will activate
on Tezos Mainnet on block <a href="https://tzkt.io/2981889">#2,981,889</a>, which
is currently expected at <a href="https://tzstats.com/election/head">early hours of December 19th
<span class="caps">CET</span></a>. <a href="https://tezos.gitlab.io/releases/version-15.html#version-15-1">Octez
v15.1</a>
will be the minimal compatible version required in order to
participate in the consensus. As such, we recommend you take
sufficient time to test this new Octez version to avoid surprises
closer to the activation date.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:simple">
<p>This is, of course, a very simplified version of the
story. Tezos uses a particular Irmin back-end, <code>irmin-pack</code>,
developed by Tarides and catering for our special needs. If you
are interested in learning more about it — <a href="https://tarides.com/blog/2020-09-01-introducing-irmin-pack">this blog
post</a>
is a good entrypoint. <a class="footnote-backref" href="#fnref:simple" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:time">
<p>Or a bit over 50 months or 4 years and something in human
time. But, as Tezos reminds ourselves on each protocol upgrade
activation… the natural unit of time in a blockchain is the
block height or level. <a class="footnote-backref" href="#fnref:time" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:ohdear">
<p>Oh dear! if they are not. <a class="footnote-backref" href="#fnref:ohdear" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:pruning">
<p>Aka as <em>garbage collection</em> of the context
storage. However, we prefer to rather use pruning here because, unlike
the case of good old, memory management, garbage collection, the data
in question is <em>definitely not</em> garbage. Even if the technique is
indeed a garbage collection mechanism — as we discuss <a href="##gc">later
on</a>. <a class="footnote-backref" href="#fnref:pruning" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:monitor">
<p>If <a href="##Figure-1">Figure 1</a> caught your attention and you want
to know more on how to monitor your Octez Tezos node, we recommend
you <a href="https://research-development.nomadic-labs.com/octez-metrics.html">add this recent
article</a>
to your reading list. <a class="footnote-backref" href="#fnref:monitor" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:consensus">
<p>For <a href="https://tezos.gitlab.io/active/consensus.html#validator-selection-staking-balance-active-stake-and-frozen-deposits">consensus
reasons</a>,
the Octez node must keep the information from at least 5 cycles,
as prescribed by the <code>PRESERVED_CYCLES</code> protocol constant. To ease
bakers’ daily operations, the node keeps one additional cycle by
default — that is, a total of 6 cycles. Until this 6-cycle
window is reached, no context pruning will be performed. <a class="footnote-backref" href="#fnref:consensus" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
</ol>
</div>A faster and more scalable Tezos: A sneak peak at the Mumbai proposal2022-12-08T18:00:00+01:002022-12-08T18:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2022-12-08:/mumbai-preview.html<p>A joint post from Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda <span class="amp">&</span> Functori.</p><p><em>This is a joint post from Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda <span class="amp">&</span> Functori.</em></p>
<p>With the Lima upgrade set to activate on Dec 19, at block <a href="https://tzkt.io/2981889">2,981,889</a>, let’s look ahead to what’s next for Tezos. For our 13th upgrade proposal to come in 2023, we will journey from Lima in the Pacific to the gateway of India in the Arabian Sea: Mumbai.</p>
<p>There are 3 major innovations coming in the Mumbai upgrade proposal, in addition to other improvements. We recently wrote about how <a href="https://research-development.nomadic-labs.com/smart-rollups-are-coming.html">smart rollups are coming</a> to Tezos, and with Mumbai they officially arrive. With the capabilities that Smart Rollups provide, the rails to reach 1 million <span class="caps">TPS</span> are in place. In addition, we are introducing Epoxy—our validity rollup (aka <span class="caps">ZK</span>-rollup) solution — on testnets. And lastly, Tezos is getting faster: we are reducing the block time to 15 seconds!</p>
<h2 id="smart-rollups-activated"><a class="toclink" href="#smart-rollups-activated">Smart Rollups activated</a></h2>
<p>The activation of <a href="https://tezos.gitlab.io/alpha/smart_rollups.html">smart rollups</a> in Mumbai is the result of an intense year-long undertaking. Smart rollups offer a powerful scaling solution for Tezos, allowing anyone to deploy decentralized WebAssembly applications with dedicated computational and networking resources. Smart rollups have been available on the <a href="https://teztnets.xyz/mondaynet-about">Mondaynet</a> testnet since the Lima proposal, but have undergone a number of significant improvements in their design and implementation during the development of Mumbai.</p>
<p>First, while in Lima each rollup had a dedicated inbox, in Mumbai there is <a href="https://tezos.gitlab.io/alpha/smart_rollups.html#inputs">only one global inbox</a> shared by all rollups, which enables more use cases. Second, decentralized applications running on rollups can now retrieve information from a <a href="https://tezos.gitlab.io/alpha/smart_rollups.html?highlight=smart%20rollups#reveal-data-channel">“reveal-data channel”</a>, enabling access to data sources external to the inbox or even external to the Tezos blockchain. And third, smart rollups are forward compatible with the Data-Availability Layer (<span class="caps">DAL</span>), a data sharding feature that is in the works for improving scalability.</p>
<h2 id="epoxy-validity-rollups-on-mondaynet-testnet"><a class="toclink" href="#epoxy-validity-rollups-on-mondaynet-testnet">Epoxy: Validity Rollups on Mondaynet testnet</a></h2>
<p>Validity rollups (also known as <span class="caps">ZK</span>-rollups) are coming to Tezos starting with Mumbai (in addition to Smart Rollups). This will be introduced gradually over multiple upgrades. Our implementation of validity rollups is named <a href="https://gitlab.com/tezos/tezos/-/merge_requests/6855">Epoxy</a>, and allows for instant finality due to <span class="caps">SNARK</span>’s proof-of-validity, as opposed to the “refutation game” used by <a href="https://tezos.gitlab.io/alpha/smart_rollups.html">Optimistic Rollups</a>. Epoxy allows for applications to be built on Tezos which do not necessarily need to rely on Tezos’ Layer 1 to publish and distribute data, but rather can implement their own data availability layers. This will enable building privacy-preserving solutions which require keeping sensitive information (reified as part of their state or transactions) private from the public Layer 1.</p>
<p>Epoxy will be soon available on <a href="https://teztnets.xyz/mondaynet-about">Mondaynet</a> and allows Tezos’ economic protocol to host new validity rollups written as circuits for the <a href="https://gitlab.com/nomadic-labs/cryptography/privacy-team/">aPlonK</a> proving system. The first instance that will be originated on Mondaynet is a transactional rollup handling tickets – that is providing a similar functionality to what <a href="https://tezos.gitlab.io/active/transaction_rollups.html">TORUs</a> do, but benefiting from instant validation.</p>
<h2 id="pipelining-is-fully-deployed-giving-us-15-second-block-times"><a class="toclink" href="#pipelining-is-fully-deployed-giving-us-15-second-block-times">Pipelining is fully deployed, giving us 15 second block times!</a></h2>
<p>The validation pipelining project has been transforming the protocol’s internal business logic, and different milestones have been deployed with the <a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html#first-steps-towards-pipelined-block-validation">Kathmandu</a> and <a href="https://research-development.nomadic-labs.com/lima-preview.html#more-pipelining-for-a-faster-blockchain">Lima</a> protocol upgrades. Our end goal was clear from the start: streamlining the internal validation process without compromising the security of the chain.</p>
<p>Mumbai brings the final installment of this project, which delivers optimizations in the protocol by separating the <a href="https://tezos.gitlab.io/alpha/validation.html">validation from application of blocks and operations</a>. This allows blocks and operations to be propagated faster, giving us confidence to reduce block times. With Mumbai, we halve the minimal time between blocks from 30s to 15s.</p>
<p>To summarize, in Kathmandu we first focused on applying this new validation discipline to manager operations—e.g. transactions and smart contract calls, which are the ones which consume the most resources. In Lima, we extended our scope to all operations supported by the Tezos economic protocol, and to how blocks are handled when nodes receive them. And now in Mumbai, pipelining is brought to the baker, so the block payload (smart contracts, etc.) is not executed before the proposed block is propagated—the block must only go through a lightweight validity check (see our <a href="https://research-development.nomadic-labs.com/lima-preview.html#more-pipelining-for-a-faster-blockchain">previous discussions</a> on pipelining for more details).</p>
<p>Since baking will only consist of sorting and validating selected operations and signing the block, the delay in block propagation is reduced significantly, giving us sufficient confidence to safely halve minimal block time to 15 seconds.</p>
<p>This is the second time Tezos reduces block times via on-chain protocol amendment. They were first cut in half from 1 minute to 30 seconds with the adoption of the <a href="https://research-development.nomadic-labs.com/faster-finality-with-emmy.html">Emmy*</a> consensus algorithm, activated in Granada. And now with Mumbai, they are halved again to 15 seconds. We will continue to streamline the implementation of the Tezos economic protocol and the Octez suite to bring faster block times.</p>
<h2 id="other-improvements"><a class="toclink" href="#other-improvements">Other Improvements</a></h2>
<ul>
<li>
<p><em>Implicit account tickets:</em> With Mumbai, implicit accounts can receive, store and
<a href="https://gitlab.com/tezos/tezos/-/merge_requests/6108">send tickets to other implicit accounts</a>
and originated contracts. In the previous protocols, <a href="https://opentezos.com/tezos-basics/tickets/">tickets</a> could only be stored in contract/<span class="caps">TORU</span> storage or sent to other contracts/TORUs. With this upgrade, contracts may also send tickets to implicit accounts so that they can store and send to other entities later; or tickets can be exchanged between implicit accounts.</p>
</li>
<li>
<p><em>RPCs for ticket balances:</em> We added two <span class="caps">RPC</span> endpoints to improve the visibility of ticket ownership.</p>
<ul>
<li>
<p><code><contract-id>/all_ticket_balances</code> endpoint that returns a complete list of tickets owned by a given contract by scanning the contract’s storage. It can only be used for originated contracts. In conjunction with the <a href="https://www.marigold.dev/post/ticket-updates-in-receipts">ticket update field in transaction receipt</a>, this enables indexers to reconstruct a “global ticket table” that keeps track of all ticket ownership</p>
</li>
<li>
<p><code><contract-id>/ticket_balance</code> endpoint that returns the given contract’s balance of the ticket with specified ticketer, content type, and content. It can be used for implicit accounts, originated contracts, and rollups. This can be used to double-check ticket ownership.</p>
</li>
</ul>
</li>
<li>
<p><em>New Michelson operations:</em> First, we extended the Michelson opcodes <code>AND, OR, XOR, NOT, LSL</code> and <code>LSR</code> to support <a href="https://gitlab.com/tezos/tezos/-/merge_requests/6055">logical operations on <code>bytes</code></a>. Their semantics are similar to those on <code>nat</code>. Second, we added an opcode to <a href="https://gitlab.com/tezos/tezos/-/merge_requests/6681">convert between <code>bytes</code> and <code>nat</code></a>.</p>
</li>
</ul>
<h2 id="now-its-your-turn"><a class="toclink" href="#now-its-your-turn">Now it’s your turn</a></h2>
<p>As it was the case in 2022, we are pushing to make 2023 a year of ground-breaking advances, and Mumbai brings to life the monumental work initiated in Kathmandu and continued with Lima. Of course, the technical advances in smart rollups and validity rollups (Epoxy) will only really show their power once new applications are built with them. So we encourage the Tezos developer community to build with these technologies and show everyone what is possible.</p>
<p>To learn more about Smart Rollups:</p>
<ul>
<li><a href="https://research-development.nomadic-labs.com/smart-rollups-are-coming.html">The road to a million <span class="caps">TPS</span> (and beyond): Smart rollups are coming</a> (blog post).</li>
<li><a href="https://research-development.nomadic-labs.com/next-generation-rollups.html">Why the next generation of optimistic rollups are a game-changer for Tezos</a> (blog post).</li>
<li><a href="https://www.youtube.com/watch?v=SRDlaAhhKBY">Scaling Tezos with rollups</a> (presentation by developers).</li>
<li>Look to <a href="https://research-development.nomadic-labs.com/blog.html">our blog</a> for articles on Epoxy in the coming weeks.</li>
</ul>
<p>Anyone interested in getting started with building a smart rollup node can reach out to <a href="mailto:contact@nomadic-labs.com">contact@nomadic-labs.com</a>.</p>Improving Randomness in Tezos with Verifiable Delay Functions2022-11-21T18:00:00+01:002022-11-21T18:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2022-11-21:/verifiable-delay-functions.html<p>Verifiable Delay Functions Strengthen Tezos’ Randomness Generation</p><p>In this article we explain how a novel cryptographic primitive called a Verifiable Delay Function (<span class="caps">VDF</span>) improves random seed generation in Tezos. This novel feature was <a href="https://research-development.nomadic-labs.com/announcing-tezos-11th-protocol-upgrade-proposal-kathmandu.html">announced in July</a>, and went live with the <a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html#improved-randomness-using-verifiable-delay-functions">activation of the Kathmandu protocol</a> in September.</p>
<h2 id="randomness-in-a-blockchain"><a class="toclink" href="#randomness-in-a-blockchain">Randomness in a blockchain</a></h2>
<p>Tezos is a blockchain that relies on a consensus mechanism based on <a href="https://tezos.gitlab.io/active/proof_of_stake.html">proof-of-stake</a>, which requires that participants be chosen at random for each cycle. These participants, or delegates, are appointed to bake and endorse blocks for which they earn rewards. Since a delegate’s stake can fluctuate, randomness is used as a mechanism to fairly distribute baking and endorsing rights based on a <a href="https://tezos.gitlab.io/active/proof_of_stake.html#slot-selection">discrete probability distribution</a> in proportion to the delegate’s stake.</p>
<p>However, since a blockchain is supposed to be deterministic (everything should be re-computable), how can we achieve randomness? Further discussion on why randomness is challenging in a blockchain can be found <a href="https://our.status.im/two-point-oh-randomness/">here</a>. It is important to be able to produce random numbers in a way that can’t be gamed, since participants have a strong incentive to cheat in order to get more rights and thus more rewards.</p>
<p>Various approaches to getting random numbers in blockchains have been devised. Generally they either rely on external submissions of values that are provably random, or they combine multiple values produced by participants into a single random value. These range from <a href="https://hackernoon.com/generating-randomness-in-blockchain-verifiable-random-function-ft1534ud"><span class="caps">VRF</span> (Verifiable Random Function)</a> solutions, including <a href="https://chain.link/vrf">Chainlink’s <span class="caps">VRF</span></a> to the <a href="https://github.com/randao/randao"><span class="caps">RANDAO</span></a> algorithm used in Ethereum. The <a href="https://tezos.com/whitepaper.pdf">Tezos whitepaper</a> describes our <span class="caps">MPC</span> (multiparty computation) commit-and-reveal scheme. While it precedes Ethereum’s <span class="caps">RANDAO</span>, we have subsequently adopted the term since the algorithms are similar.<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup></p>
<h2 id="randomness-in-tezos-with-randao"><a class="toclink" href="#randomness-in-tezos-with-randao">Randomness in Tezos with <span class="caps">RANDAO</span></a></h2>
<p>Prior to Kathmandu, Tezos only used <span class="caps">RANDAO</span> (that is, our variant of it). The basic idea is to incentivize participants to submit random values, or “nonces”, that we combine on-chain into a “random seed”. When selected to submit one of these values, delegates stand to lose part of their rewards for a given cycle if they fail to do so.</p>
<p><span class="caps">RANDAO</span> works as follows: to compute a seed for cycle <code>n</code> i.e. <code>seed</code><sub><code>n</code></sub>, we go back in time to cycle <code>n-2-PRESERVED_CYCLES</code> (<code>PRESERVED_CYCLES</code> is a <a href="https://tezos.gitlab.io/active/proof_of_stake.html#proof-of-stake-parameters">protocol constant</a>, set to 5 at the time of writing) where we asked delegates to submit <em>commitments</em> to random values <code>commitments</code><sub><code>{n-2-PRESERVED_CYCLES}</code></sub>. The commitments are hashes of random values. Submitting hashes rather than the random values themselves at this stage is done to prevent an adversary from adaptively choosing its random value to its advantage, for instance to achieve more baking rights and thereby more rewards.
In the following cycle, <code>n-1-PRESERVED_CYCLES</code>, the delegates were asked to submit their random values <code>nonces</code><sub><code>{n-1-PRESERVED_CYCLES}</code></sub>. If the hash of these values equals the corresponding commitment, we store them. The seed is finally computed at the end of the cycle as the hash of the previous target seed and the stored revealed values (in the order in which they were committed): <code>seed</code><sub><code>n</code></sub> <code>= Randao</code><sub><code>{n-1-PRESERVED_CYCLES}</code></sub> <code>= Hash(seed</code><sub><code>{n-1}</code></sub><code>, nonces</code><sub><code>{n-1-PRESERVED_CYCLES}</code></sub><code>)</code>.
As long as one of the participants is honest (that is, they submit an honestly generated random value), and the hash function is secure, so is the scheme.</p>
<h2 id="last-revealers-advantage"><a class="toclink" href="#last-revealers-advantage">Last revealer’s advantage</a></h2>
<p>As previously stated, this algorithm is deemed secure if at least one honest member participates. However, the algorithm isn’t perfect as it can offer an advantage to the last revealer. The last revealer, seeing all revealed values, can choose whether to reveal their commitment or not, in effect choosing a seed between two potential values, resulting in “biasable randomness”. In truth, the advantage can extend to that last <em>n</em> revealers if they collude.</p>
<p>This potential for bias can be mitigated in various ways. Examples include using a <a href="https://ethresear.ch/t/making-the-randao-less-influenceable-using-a-random-1-bit-clock/2566">1 bit clock</a> and <a href="https://ethresear.ch/t/avalanche-randao-a-construction-to-minimize-randao-biasability-in-the-face-of-large-coalitions-of-validators/4248">Avalanche’s <span class="caps">RANDAO</span> construction</a>.</p>
<p>The other main alternative to <span class="caps">RANDAO</span> for generating unbiased randomness is <em>Threshold Relay</em> which is used in <span class="caps">DFINITY</span>. However, it relies on an expensive distributed key generation mechanism ( <span class="math">\(\mathcal{O}(n^2)\)</span> messages for n members) and high liveness requirements—the beacon may stall if <a href="https://ethresear.ch/t/minimal-vdf-randomness-beacon/3566">even 15% of honest players</a> go offline.</p>
<h2 id="unbiased-randomness-with-verifiable-delay-functions"><a class="toclink" href="#unbiased-randomness-with-verifiable-delay-functions">Unbiased randomness with Verifiable Delay Functions</a></h2>
<p>Currently, the most sophisticated method to remove this bias in <span class="caps">RANDAO</span> is by using a <a href="https://eprint.iacr.org/2018/712.pdf"><em>Verifiable Delay Function (<span class="caps">VDF</span>)</em></a>. VDFs are a novel cryptographic primitive that allow a verifier to check that a value was computed by a prover as a result of a long computation. Appending a <span class="caps">VDF</span> phase after our current <span class="caps">RANDAO</span> scheme would prevent any participant from anticipating what the next seed might be, because the fast hash function is replaced by a (provably) long computation. While VDFs are state-of-the-art cryptographic primitives they were recently integrated into Tezos with the <a href="https://research-development.nomadic-labs.com/kathmandu-is-live.html">activation of the Kathmandu protocol upgrade</a>. Ethereum is planning to add <span class="caps">VDF</span> for random number generation in the future<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup>.</p>
<p>For the implementation of the underlying cryptographic primitive, we adapt the <a href="https://gitlab.com/nomadic-labs/cryptography/ocaml-chia-vdf"><span class="caps">VDF</span> library</a> developed by <a href="https://www.chia.net/">Chia</a>. Chia however doesn’t use it for random number generation, but rather for creating time slots in its consensus algorithm, to prove some amount of elapsed time.</p>
<h2 id="seed-generation-with-vdf"><a class="toclink" href="#seed-generation-with-vdf">Seed Generation with <span class="caps">VDF</span></a></h2>
<p>The diagram below illustrates the steps in creating a random seed with <span class="caps">VDF</span>.</p>
<figure align="center">
<img src="/images/vdf_computation.png" alt="VDF diagram" width="60%">
<figcaption align="center"> </figcaption>
</figure>
<p><span class="caps">RANDAO</span> with <span class="caps">VDF</span> works as follows:</p>
<ol>
<li>In cycle # <code>n-2-PRESERVED_CYCLES</code> the parties publish commitments to random values;</li>
<li>In cycle # <code>n-1-PRESERVED_CYCLES</code> the nonces are revealed, but the revelation phase is now shortened from the entirety of the cycle to only the first <code>NONCE_REVELATION_THRESHOLD</code> blocks;</li>
<li>At block <code>NONCE_REVELATION_THRESHOLD</code>, the <span class="caps">RANDAO</span> output is then computed and used to instantiate a hidden order group (an algebraic group whose number of elements is unknown), and a challenge to compute a <span class="caps">VDF</span> solution for. Any party can query the information needed to compute the <span class="caps">VDF</span> solution through RPCs;</li>
<li>The first baker to disclose the correct solution before the end of cycle <code>n-1-PRESERVED_CYCLES</code> receives a small tip;</li>
<li>The random seed for cycle <code>n</code> is then computed as either the hash of the <span class="caps">RANDAO</span> output and the <span class="caps">VDF</span> solution, if a valid <span class="caps">VDF</span> solution is received, or the <span class="caps">RANDAO</span> output otherwise.</li>
</ol>
<p>That is, <code>seed</code><sub><code>n</code></sub> <code>= Hash(Randao</code><sub><code>{n-1-PRESERVED_CYCLES}</code></sub><code>, VDF</code><sub><code>{n-1-PRESERVED_CYCLES}</code></sub><code>)</code>, or <code>seed</code><sub><code>n</code></sub> <code>= Randao</code><sub><code>{n-1-PRESERVED_CYCLES}</code></sub> if no valid <span class="caps">VDF</span> was submitted in cycle <code>n-1-PRESERVED_CYCLES</code>.</p>
<p>Note that even with a valid <span class="caps">VDF</span> output we still use the time-tested <span class="caps">RANDAO</span> when computing the randomness seed, as shown in step 5, in order to make the whole system more robust. Indeed, while the <span class="caps">VDF</span> output can be considered as unpredictable it may not be random enough, hashing it guarantees the seed randomness. See the <a href="https://tezos.gitlab.io/active/randomness_generation.html">Tezos documentation</a> and <a href="https://medium.com/iovlabs-innovation-stories/verifiable-delay-functions-8eb6390c5f4">this Medium article</a> for more details on <span class="caps">VDF</span>.</p>
<h2 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h2>
<p>The regular upgrade cycles of the Tezos protocol mean the latest advances in cryptographic research can be quickly implemented into the protocol. While the <span class="caps">RANDAO</span> protocol is used in a number of blockchains, and while VDFs are widely discussed, Tezos is currently the only blockchain which uses it for randomness generation, making random seed generation in Tezos highly robust compared to its competitors.</p>
<p>And that is the ethos of Tezos: to continuously monitor new research and technological developments, evaluate what’s best suited to be adopted, and integrate it quickly.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>Note that in some versions of <span class="caps">RANDAO</span> <a href="https://revelry.co/insights/blockchain/critical-randao-vulnerability/">there is a vulnerability</a> when using <span class="caps">XOR</span> to combine nonces. Tezos’ <span class="caps">RANDAO</span> hashes the nonces together so this vulnerability doesn’t exist. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>Ethereum is working on a hardware solution, while we currently use a software one (a hardware solution doesn’t exist yet). Note that Tezos and Ethereum are advisory partners in the <a href="https://www.vdfalliance.org/partners"><span class="caps">VDF</span> Alliance</a>. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>The road to a million TPS (and beyond): Smart rollups are coming2022-11-16T14:00:00+01:002022-11-16T14:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2022-11-16:/smart-rollups-are-coming.html<p>A new reference manual is out, and we are looking to onboard early adopters with relevant use cases.</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: Smart rollups are feature complete, a reference manual is ready, and we are looking to onboard early adopters with relevant use cases.</strong></p>
<p>The next generation of optimistic rollups on Tezos are nearing their inclusion in a protocol upgrade proposal, and we have interesting news to share.</p>
<p>If you haven’t read our <a href="https://research-development.nomadic-labs.com/next-generation-rollups.html">introduction to smart rollups</a>, the elevator pitch is that they are more than just smart contract-enabled optimistic rollups with high scalability.</p>
<p>They allow anyone to program and deploy their own interoperable Layer 2 solution in a number of popular programming languages, using a general-purpose infrastructure built for decentralization and security.</p>
<p>Fancy an <span class="caps">EVM</span>-compatible “sidechain” in the Tezos ecosystem? This can be implemented using the new rollup infrastructure. We are already exploring this and will ramp up our work once smart rollups are live on Tezos Mainnet.</p>
<p>Smart rollups also enable an entirely new way to write decentralized applications on Tezos. In addition to using smart contracts on Layer 1, developers will be able to implement a dapp as a rollup, though this is mostly relevant for users requiring high throughput and/or intense computation. </p>
<p>Doing so effectively makes the smart rollup work like an app-chain – a blockchain dedicated to a single application – but one that draws on the time-tested security model of the Tezos network.</p>
<p>Finally, let’s not forget scaling: smart rollups play a central role in our mission to reach one million transactions per second (<span class="caps">TPS</span>) on Tezos, as <a href="https://youtu.be/SRDlaAhhKBY?t=184">announced at the TezDev Paris conference</a> earlier this year. </p>
<p>Note that the million <span class="caps">TPS</span> goal is one we have set for ourselves for the short term. In the long term, due to the <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html#the-best-of-both-worlds">combination of horizontal and vertical scaling</a>, there is virtually no limit to the scaling potential of Tezos rollups.</p>
<h2 id="enshrined-what"><a class="toclink" href="#enshrined-what">Enshrined? What?</a></h2>
<p>The key to the flexibility, decentralization, and security of these next-generation rollups is that they are implemented at the protocol level, making them a type of <em>enshrined</em> rollups. </p>
<p>Googling <a href="https://www.google.com/search?q=enshrined">“enshrined”</a> will get you the following definitions</p>
<ol>
<li><em>“place (a revered or precious object) in an appropriate receptacle”</em> </li>
<li><em>“preserving something in a form that ensures it will be protected and respected”</em>.</li>
</ol>
<p>And that is what enshrined rollups are about. Instead of deploying rollups as smart contracts like it’s done on, e.g., Ethereum, they are officially recognized by the Tezos protocol as special entities with particular privileges and features. </p>
<p>You could even argue that such tight integration with the underlying blockchain makes enshrined rollups a feature of Layer 1, rather than a Layer 2 solution.</p>
<p>Enshrined rollups also represent a social contract between protocol developers and the community. By design, they are equally available to everyone. And, if adopted by the community through a protocol proposal, they will be maintained and continuously improved in ways that benefit the Tezos network as a whole. </p>
<p>On the technical side, the benefits of enshrined rollups are significant</p>
<ul>
<li>It generally enables complex features that are at best difficult and often impossible to implement using a smart contract-based rollup.</li>
<li>Rollup-related Layer 1 activity can be made much more computation/gas efficient.</li>
<li>Common infrastructure enables standardized and efficient communication with Layer 1 and between rollups, even with different execution environments.</li>
<li>Special data solutions can serve these rollups, so rollup-data doesn’t consume excessive storage and bandwidth on Layer 1.</li>
</ul>
<h2 id="ready-for-data-availability-solutions"><a class="toclink" href="#ready-for-data-availability-solutions">Ready for data availability solutions</a></h2>
<p>The ability to create tailored data solutions is particularly important for scalability.</p>
<p>While smart rollups are deeply integrated to the Layer 1 protocol, we want to avoid having Layer 1 becoming a bottleneck, as described in our <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html#next-level-scaling">post on scaling strategy</a>. Trying to reach our million <span class="caps">TPS</span> goal would quickly fill up Layer 1 blocks, and bandwidth on the network would become an issue. </p>
<p>To tackle this challenge, we are implementing a general and flexible mechanism for providing arbitrary data to a rollup without going through Layer 1. The mechanism can be utilized in different ways, but we expect to see two approaches:</p>
<p>A <em>Data Availability Committee</em> (<span class="caps">DAC</span>) is an off-chain mechanism to keep data available for verification, which relies on a trusted set of data providers. This presents a compromise with regards to decentralization, and unless a user has specific needs for control of rollup data – for instance in a private solution – the use of DACs will likely be transitional.</p>
<p>The more decentralized approach is a <em>Data Availability Layer</em> (<span class="caps">DAL</span>), which serves data like a <span class="caps">DAC</span>, but is a “<a href="https://en.wikipedia.org/wiki/Public_good_(economics)">public good</a>” provided by the protocol itself: a dedicated rollup data layer for Tezos. Such a layer is in development, and will be included in a future protocol upgrade proposal.</p>
<h2 id="new-reference-manual-for-dapp-developers"><a class="toclink" href="#new-reference-manual-for-dapp-developers">New reference manual for dapp developers</a></h2>
<p>The (first) execution environment for smart rollups is WebAssembly (<span class="caps">WASM</span>), which was chosen for at least two reasons. First, <span class="caps">WASM</span> is designed for fast execution, and second, <span class="caps">WASM</span> is becoming a broadly adopted compilation target, which means you will be able to write your “smart rollup contracts” in a large set of popular programming languages.</p>
<p>We invite interested readers to explore <a href="https://tezos.gitlab.io/alpha/smart_rollups.html">our new reference manual</a>, which demonstrates how to concretely implement a dapp as a smart rollup. </p>
<p>The manual covers </p>
<ul>
<li>The generic optimistic rollup infrastructure developed for Tezos.</li>
<li>The life-cycle of a <span class="caps">WASM</span> smart rollup.</li>
<li>The software environment in which the smart rollup program is executed, including the <span class="caps">API</span> smart rollups can use.</li>
<li>How Rust can be used to implement a simple smart rollup.</li>
<li>How this simple smart rollup can be deployed on Tezos.</li>
</ul>
<p>We have also released a Rust module that defines the low-level <span class="caps">API</span> that is made available to developers. We would love to see what kind of safe and efficient <span class="caps">API</span> Rust developers can come up with.</p>
<p>Of course, the language used is not limited to Rust. Any programming language that compiles to <span class="caps">WASM</span> and does not assume a browser as its sole execution environment can be used to implement smart rollups.</p>
<h2 id="start-your-wasm-engines"><a class="toclink" href="#start-your-wasm-engines">Start your <span class="caps">WASM</span> engines</a></h2>
<p>The coming months are about making sure the Tezos ecosystem is off to a running start, once smart rollups are activated on Mainnet. It involves two important tasks:</p>
<p><strong>Ecosystem testing:</strong> Smart rollups on <a href="https://teztnets.xyz/mondaynet-about">Mondaynet</a> are now feature complete. Things like APIs can still change, so (minor) breaking changes are possible, but there will be no paradigm or design changes. It’s time to start experimenting, and we are looking to onboard early adopters.</p>
<p>If you are part of a project/organization that requires the scalability or flexibility that smart rollups offer, don’t wait for Mainnet activation. <a href="https://adoption-support.nomadic-labs.com/contact/">Get in touch now</a>, and get help with building your solution while helping us ensure smart rollups are optimized for our users’ needs upon Mainnet activation.</p>
<p><strong>Tweaking and tuning:</strong> We are working on making sure the infrastructure for <span class="caps">WASM</span> smart rollups can handle the high throughput we have committed to reaching on Tezos. For instance, the rollup node that will be shipped with Octez will make good use of the performances of the <a href="https://github.com/wasmerio/wasmer"><span class="caps">WASMER</span></a> execution engine.</p>
<p>Provided that no blocking issues are uncovered in this final testing phase, activation of smart rollups on Mainnet will be included in the ‘M’ protocol proposal in early 2023.</p>
<p>We look forward to exploring the power of this new rollup engine, and to sharing our experiences with the community. Stay tuned!</p>
<p><strong>Learn more about smart rollups:</strong></p>
<ul>
<li>
<p><a href="https://research-development.nomadic-labs.com/next-generation-rollups.html">Why the next generation of optimistic rollups are a game-changer for Tezos </a> (blog post)</p>
</li>
<li>
<p><a href="https://www.youtube.com/watch?v=SRDlaAhhKBY">Scaling Tezos with rollups</a> (presentation by developers)</p>
</li>
</ul>Announcement: New Names for Octez Executables in v15.02022-10-11T14:00:00+02:002022-10-11T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-10-11:/filename-renaming-octez-v15.html<p>Octez v15.0 will begin to enforce the Octez brand on the implementation</p><p>In Octez v15.0, the next major release of the Octez suite, the names
of executable binaries will change:</p>
<ul>
<li>the <code>tezos-</code> prefix becomes <code>octez-</code>, and;</li>
<li>the protocol number (e.g., the <code>014</code> in
<code>tezos-tx-rollup-node-014-PtKathma</code>) is removed from
protocol-specific executables — baker, accuser, Layer 2 nodes and
clients, etc.</li>
</ul>
<p>These changes have already been implemented in the Octez <code>master</code>
branch, and will be enforced from its first release candidate,
<code>v15.0~rc1</code>.</p>
<p>The prefix change from <code>tezos-</code> to <code>octez-</code> is enacted for consistency
with the name we have given to <strong>this</strong> specific implementation of
Tezos — the motivation for these changes was described in <a href="https://research-development.nomadic-labs.com/announcing-octez.html">this blog
post</a>
when Octez was announced to the world. This step is meant to further
clarify that Octez executables are <em>one particular implementation of
Tezos — that is, of everything but the economic protocol itself</em>.</p>
<p>Removing protocol numbers will make it clearer that they are not
related to the Octez version numbers. Indeed, some users were confused
when Octez adopted the use of version numbers, as they eventually grew
to match protocol numbers (although this was the result of a
fortuitous coincidence).</p>
<p>Furthermore, removing the protocol number will help clarify why
different versions of protocol-specific binaries are shipped with
Octez releases, making instructions (specially around protocol
upgrades) more precise and less ambiguous. Now, those executables only
contain the name of the protocol they target, which should make it
clearer that they are a particular implementation, which is part of a
specific version of Octez.</p>
<p>Here is the full list of renamings:</p>
<ul>
<li><code>tezos-node</code> becomes <code>octez-node</code>.</li>
<li><code>tezos-client</code> becomes <code>octez-client</code>.</li>
<li><code>tezos-admin-client</code> becomes <code>octez-admin-client</code>.</li>
<li><code>tezos-signer</code> becomes <code>octez-signer</code>.</li>
<li><code>tezos-codec</code> becomes <code>octez-codec</code>.</li>
<li><code>tezos-protocol-compiler</code> becomes <code>octez-protocol-compiler</code>.</li>
<li><code>tezos-proxy-server</code> becomes <code>octez-proxy-server</code>.</li>
<li><code>tezos-validator</code> becomes <code>octez-validator</code>.</li>
<li><code>tezos-accuser-014-PtKathma</code> becomes <code>octez-accuser-PtKathma</code>.</li>
<li><code>tezos-accuser-015-PtLimaPt</code> becomes <code>octez-accuser-PtLimaPt</code>.</li>
<li><code>tezos-accuser-alpha</code> becomes <code>octez-accuser-alpha</code>.</li>
<li><code>tezos-baker-014-PtKathma</code> becomes <code>octez-baker-PtKathma</code>.</li>
<li><code>tezos-baker-015-PtLimaPt</code> becomes <code>octez-baker-PtLimaPt</code>.</li>
<li><code>tezos-baker-alpha</code> becomes <code>octez-baker-alpha</code>.</li>
<li><code>tezos-tx-rollup-client-014-PtKathma</code> becomes <code>octez-tx-rollup-client-PtKathma</code>.</li>
<li><code>tezos-tx-rollup-client-015-PtLimaPt</code> becomes <code>octez-tx-rollup-client-PtLimaPt</code>.</li>
<li><code>tezos-tx-rollup-client-alpha</code> becomes <code>octez-tx-rollup-client-alpha</code>.</li>
<li><code>tezos-tx-rollup-node-014-PtKathma</code> becomes <code>octez-tx-rollup-node-PtKathma</code>.</li>
<li><code>tezos-tx-rollup-node-015-PtLimaPt</code> becomes <code>octez-tx-rollup-node-PtLimaPt</code>.</li>
<li><code>tezos-tx-rollup-node-alpha</code> becomes <code>octez-tx-rollup-node-alpha</code>.</li>
</ul>
<h2 id="adapting-to-these-changes"><a class="toclink" href="#adapting-to-these-changes">Adapting to these changes</a></h2>
<p>We understand that these are <em>breaking</em> changes, which impact possibly
years of existing deployment infrastructure and documentation. With
this in mind, we have included in Octez v15.0 the following provisions
for maintaining backwards compatibility.</p>
<p>When building Octez v15.0 from sources using <code>make</code>, symbolic links
will be created from the old names to the new names. The list of
symbolic links is exactly the list given above. For instance, a
symbolic file named <code>tezos-baker-014-PtKathma</code> is created, which
targets <code>octez-baker-PtKathma</code>. This way, the changes brought in by
Octez v15 should be compatible with, e.g., existing deployment scripts
which rely on the explicit names of previous Octez versions’ binaries.</p>
<p>In the case of Octez Docker images, the transition is slightly less
smooth: When specifying the <code>PROTOCOL</code> environment variable to choose
which baker and accuser to run, the protocol number must not be
included. For instance, <code>PROTOCOL=PtKathma</code> must be used instead of
<code>PROTOCOL=014-PtKathma</code>.</p>
<p>Additionally, there are no symbolic links from the old names to the
new names in the Docker images themselves. However, the entrypoint
script accepts both <code>tezos-</code> and <code>octez-</code> prefixed commands. The old
<code>tezos-node</code> and <code>tezos-baker</code> commands should still work as before,
although it is now strongly recommended to use the newly added
respective replacement commands, <code>octez-node</code> and <code>octez-baker</code>, instead.</p>
<p>In the case of Octez executables downloaded from their <a href="https://gitlab.com/tezos/tezos/-/releases">release page
on GitLab</a>, these static
binaries use the new naming conventions. Users will need to either
adopt the new names, or create symbolic links themselves.</p>
<p>Note that the old naming convention is now formally <em>deprecated</em>. In a
future major release, possibly Octez v16.0, we plan to remove symbolic
links as well as the old names for the commands in the Docker image
entrypoint. It is thus a good idea to try to migrate to the new naming
conventions before Octez 16.0 is out!</p>
<p>If you are unsure about what needs to be done on your end, reach out
to us on the <a href="https://tezos-baking.slack.com/">Tezos baking Slack</a> or
feel free to contact <a href="https://adoption-support.nomadic-labs.com/contact/">the Nomadic Labs support
team</a>.</p>Announcing Tezos’ 12th protocol upgrade proposal, “Lima”2022-10-10T16:00:00+02:002022-10-10T16:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-10-10:/announcing-tezos-12th-protocol-upgrade-proposal-lima.html<p>A joint post from Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda, Functori <span class="amp">&</span> Tweag.</p><p><em>This is a joint post from Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda, <span class="amp">&</span> Functori.</em></p>
<p>We’re happy to announce that Lima, our next Tezos protocol proposal,
is ready. As usual, Lima’s ‘true’ name is its hash, which is
<code>PtLimaPtLMwfNinJi9rCfDPWea8dFgTZ1MeJ9f1m2SRic6ayiwW</code>.</p>
<p>The main features of this protocol upgrade proposal are:</p>
<ul>
<li><a href="https://research-development.nomadic-labs.com/lima-preview.html#more-pipelining-for-a-faster-blockchain">More
pipelining</a>:
The work to separate validation from application of operations and
blocks continues. It will enable higher Layer 1 throughput on Tezos.</li>
<li><a href="https://research-development.nomadic-labs.com/lima-preview.html#this-ones-for-the-bakers-introducing-consensus-keys">Consensus
keys</a>:
This much requested feature lets bakers change their key for signing
blocks and consensus operations without changing the baker’s public address.</li>
<li><a href="https://research-development.nomadic-labs.com/lima-preview.html#improvements-to-tickets">Improvements to
Tickets</a>:
Ticket ownership updates are now part of transaction receipts, which
helps indexers keep track of tickets. Also, zero-amount tickets will
be deprecated.</li>
<li><a href="https://research-development.nomadic-labs.com/lima-preview.html#ghostnet-fixes">Ghostnet
fixes</a><strong>:
</strong>Two problems arose during the migration from Jakarta to Kathmandu
on the permanent ‘Ghostnet’ test network. These are fixed in Lima.</li>
<li><a href="https://research-development.nomadic-labs.com/lima-preview.html#other-changes">Liquidity Baking sunset
removed</a>:
The sunset is no longer needed, as the subsidy can be shut off with
the moving-average toggle introduced with the Jakarta upgrade.</li>
<li><a href="https://research-development.nomadic-labs.com/lima-preview.html#other-changes">Temporary Timelock
deprecation</a>:
Due to a discovered vulnerability, origination of new contracts
using Timelock are disabled while a safer mechanism is developed.</li>
</ul>
<p>To learn more about Lima’s contents, see our <a href="https://research-development.nomadic-labs.com/lima-preview.html">full preview
post</a>.</p>
<p>The <a href="https://tezos.gitlab.io/protocols/015_lima.html">changelog</a>
provides a detailed list of changes, and a general technical overview
of Lima can be found in the protocol proposal’s <a href="https://tezos.gitlab.io/protocols/015_lima.html">technical
documentation</a>.</p>
<p>Note that, if Lima is voted in by the community, upgrading to Octez
v15.0 (or later) will be necessary for participating in consensus. A
release candidate for Octez v15.0 will be published in the coming
days, and a dedicated protocol test network <code>Limanet</code> is also
scheduled to begin soon – more information about the latter test
network will be available on
<a href="https://teztnets.xyz/">https://teztnets.xyz/</a>.</p>
<h2 id="rollup-testers-wanted"><a class="toclink" href="#rollup-testers-wanted">Rollup testers wanted</a></h2>
<p>Development of <a href="https://research-development.nomadic-labs.com/next-generation-rollups.html">next-generation enshrined optimistic
rollups</a>
on Tezos is progressing steadily. Functioning rollups implemented in
Rust are currently running in our test-suite.</p>
<p>However, these rollups are not part of the Lima protocol upgrade
proposal, as more time needs to be spent on validation, testing, and
integration with ecosystem tools before they can be activated on Mainnet.</p>
<p>We highly encourage ecosystem participants to start experimenting with
these rollups on the Mondaynet testnet, where they are already activated.</p>
<p>Broad testing and feedback from the ecosystem is invaluable in our
efforts to minimize the risk of undetected issues upon Mainnet
activation. <a href="https://research-development.nomadic-labs.com/lima-preview.html#help-us-get-rollups-rolling">Read more about how to get
started</a>.</p>What to expect in Lima — our 12th protocol upgrade proposal for Tezos2022-10-05T18:00:00+02:002022-10-05T18:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-10-05:/lima-preview.html<p>A joint post from Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda <span class="amp">&</span> Functori.</p><p><em>This is a joint post from Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda <span class="amp">&</span> Functori.</em></p>
<p>With the <a href="https://research-development.nomadic-labs.com/kathmandu-is-live.html">Kathmandu upgrade successfully
activated</a>,
it’s time to look at what’s next for Tezos.</p>
<p>For our 12th upgrade proposal, we are leaving the Himalayan mountains
behind and heading to the coastal climate of Lima, capital of Peru and
one of the largest cities in the Americas.</p>
<p>But before we embark on that journey…</p>
<h2 id="an-update-on-optimistic-rollups"><a class="toclink" href="#an-update-on-optimistic-rollups">An update on optimistic rollups</a></h2>
<p>Development of <a href="https://research-development.nomadic-labs.com/next-generation-rollups.html">next-generation enshrined optimistic
rollups</a>
on Tezos is progressing steadily. Functioning rollups implemented in
Rust are currently running in our test suite.</p>
<p>These rollups will however not be part of the Lima protocol upgrade proposal, as more time needs to be spent on validation, testing, and integration with ecosystem tools before they can be activated on Mainnet.</p>
<p><strong>We highly encourage ecosystem participants to start experimenting
with these rollups on the
<a href="https://teztnets.xyz/mondaynet-about">Mondaynet</a> testnet</strong>, where
they are already activated. Broad testing and feedback from the
ecosystem is invaluable in our efforts to minimize the risk of
undetected issues upon Mainnet activation. More on this <a href="#help-us-get-rollups-rolling">further below.</a></p>
<p>Now, let’s look at the series of improvements to Tezos functionality contained in the Lima upgrade proposal.</p>
<h2 id="more-pipelining-for-a-faster-blockchain"><a class="toclink" href="#more-pipelining-for-a-faster-blockchain">More pipelining for a faster blockchain</a></h2>
<p>Our
<a href="https://gitlab.com/tezos/tezos/-/milestones/23#tab-issues">pipelining</a>
work continues to separate <em>validation</em> from <em>application</em> of
operations and blocks in order to speed up their processing. It may
not sound sexy, but this is an important part of increasing Layer 1
throughput on Tezos.</p>
<p>A quick distinction:</p>
<ul>
<li><strong>Validating</strong> means performing basic checks, such as the
cryptographic signature being valid, and that there are funds to pay
fees. This is a light and quick process.</li>
<li><strong>Applying</strong> means executing the full contents of the operation,
whether a simple transaction or a complex contract call. This can be
much more computationally intensive and hence time consuming for the node.</li>
</ul>
<p>The <a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html#first-steps-towards-pipelined-block-validation">Kathmandu
upgrade</a>
reduced the number of times manager operations are <em>applied</em> by a node
before it is propagated through the network. This minimizes the delay
introduced with each node, as blocks and operations are gossiped
through the peer-to-peer network.</p>
<p>The Lima protocol proposal extends pipelined validation to all
remaining classes of operations, and to blocks themselves. When
receiving a new block from its peers, a node will only check the
validity of the block before forwarding it to other peers, speeding up
block propagation on the network. Afterwards, the node will then
<em>apply</em> the block.</p>
<p>The upcoming step will be to extend pipelined validation to block
production itself, reducing the effort (and hence time) required for
bakers to propose new blocks. Along with further upcoming
optimizations, this opens the door to reduced block times.</p>
<h2 id="this-ones-for-the-bakers-introducing-consensus-keys"><a class="toclink" href="#this-ones-for-the-bakers-introducing-consensus-keys">This one’s for the bakers: introducing consensus keys</a></h2>
<p>We are happy to introduce <a href="https://midl-dev.medium.com/a-consensus-key-for-tezos-bakers-16a3ac8178cf">consensus
keys</a>
— a feature which has been highly requested by bakers.</p>
<p>Consensus keys allow bakers to designate a special key — separate
from the baker address key — for signing blocks and consensus
operations, such as (pre)endorsements. The proposed implementation
lets bakers change their consensus key without changing the baker’s
public address.</p>
<p>Rotating keys is generally good practice in computer security. And
this feature will be of great benefit in situations where:</p>
<ul>
<li>There are concerns about a baker’s private key having been compromised.</li>
<li>A baker using a Key Management System (<span class="caps">KMS</span>) or Hardware Security
Module (<span class="caps">HSM</span>) wishes to switch to a different setup. These generally
don’t allow key extraction.</li>
<li>There is loss of access. E.g., if a geographically remote baking
setup using <span class="caps">KMS</span>/<span class="caps">HSM</span> fails. With consensus keys, the baker can
remotely deploy a new setup under the same baking address.</li>
</ul>
<p>Hence, a baker’s delegators no longer need to actively redelegate to a
new address, which was cumbersome and required off-chain coordination,
reducing the chance of reaching all of the baker’s delegators.</p>
<p>The consensus key feature is based on contributions made by
<a href="https://gitlab.com/g.b.fefe">G.-B. Fefe</a> (anonymous contributor) and
<a href="https://gitlab.com/Ochem">Nicolas Ochem</a>. As a reward for their
involvement, invoices of respectively 15,000 and 10,000 tez are
included in the Lima proposal.</p>
<h2 id="improvements-to-tickets"><a class="toclink" href="#improvements-to-tickets">Improvements to Tickets</a></h2>
<p>We are <a href="https://www.marigold.dev/post/remove-support-for-tickets-with-zero-amount">deprecating creation, storage and transfer of zero-amount
tickets</a>. This
removes a source of inconvenience and reduces the risk of bugs in
smart contracts, but introduces a <a href="https://tezos.gitlab.io/protocols/015_lima.html#breaking-changes">breaking
change</a>
in the <code>TICKET</code> instruction.</p>
<p>Furthermore, we added ticket ownership updates to transaction
receipts. This enables indexers to maintain a table that tracks which
accounts own what tickets by traversing the receipts.</p>
<p>The change to receipts was introduced after fruitful meetings with
ecosystem actors, and we are happy to collaborate with the ecosystem
in this way and implement their valuable feedback.</p>
<p>In Lima, receipts will include this information for ticket
minting/removal within a single contract and transfers between
originated contracts. In upcoming upgrades, we will cover all
combinations of transactions between implicit accounts, originated
accounts, and rollups. A design document can be found
<a href="https://hackmd.io/@marigold-dev/SyLXJQcJs">here</a>.</p>
<h2 id="ghostnet-fixes"><a class="toclink" href="#ghostnet-fixes">Ghostnet fixes</a></h2>
<p>During the migration from Jakarta to Kathmandu on the Ghostnet test
network, two problems arose, that are fixed in Lima:</p>
<ul>
<li>
<p>The <a href="https://tezos.gitlab.io/lima/randomness_generation.html#verifiable-delay-function"><span class="caps">VDF</span>
feature</a>
activated itself with the same difficulty as on Mainnet, but cycles
on Ghostnet are ¼ of Mainnet! So it’s impossible to do the required
computation within the allocated time frame. For this reason, the
<span class="caps">VDF</span> challenge’s difficulty on Ghostnet is now set to ¼ of Mainnet’s.</p>
</li>
<li>
<p>The length of a voting period on Ghostnet was changed from two
cycles to one cycle. However, due to the way a protocol upgrade is
executed, a <em>“time until next period”</em>-counter became negative after
Kathmandu was activated, leading Ghostnet to not advance through
voting periods automatically. Therefore, a ‘force reset’ of the
voting period is scheduled for Ghostnet’s migration to Lima, should
the proposal be adopted by the community.</p>
</li>
</ul>
<h2 id="other-changes"><a class="toclink" href="#other-changes">Other changes</a></h2>
<ul>
<li><strong>Liquidity Baking sunset removed</strong>: The liquidity baking sunset
will be removed, since the subsidy can now be shut off with the
moving-average toggle <a href="https://research-development.nomadic-labs.com/announcing-tezos-10th-protocol-upgrade-proposal-jakarta.html#liquidity-baking-toggle-vote">introduced with the Jakarta
upgrade</a>.</li>
<li><strong>Temporary Timelock deprecation:</strong> <a href="https://forum.tezosagora.org/t/vulnerability-found-in-the-timelock-feature/4660">As recently
announced</a>,
a vulnerability has been discovered in the Timelock mechanism. A
safer Timelock mechanism is currently being developed. As a
preventive measure, <a href="https://tezos.gitlab.io/CHANGES.html#client">Octez
v14</a> already disabled
interaction with smart contracts using this feature. The Lima
protocol proposal complements this measure <a href="https://tezos.gitlab.io/protocols/015_lima.html#breaking-changes">by preventing the
origination of new
contracts</a>
using this functionality. This is achieved by temporarily
deprecating the <code>CHEST_OPEN</code> instruction in Michelson.</li>
</ul>
<h2 id="help-us-get-rollups-rolling"><a class="toclink" href="#help-us-get-rollups-rolling">Help us get rollups rolling!</a></h2>
<p>As mentioned initially, we are looking to have our <a href="https://research-development.nomadic-labs.com/next-generation-rollups.html">next-generation
optimistic
rollups</a>
tested extensively and integrated with ecosystem tools before
activation on Mainnet.</p>
<p>Careful and incremental integration in close cooperation with the
Tezos ecosystem is always desirable, but with optimistic rollups being
the backbone of Tezos’ scaling strategy and possibly the largest
evolutionary leap for the Tezos protocol so far, this becomes paramount.</p>
<p>In particular, the Webassembly (<span class="caps">WASM</span>) <a href="https://research-development.nomadic-labs.com/next-generation-rollups.html#the-proof-generating-virtual-machine">Proof-generating Virtual
Machine</a>
(<span class="caps">PVM</span>), which is at the heart of the system, is currently considered to
be in beta.</p>
<p>The <span class="caps">WASM</span> <span class="caps">PVM</span> included in Lima lets users originate a rollup on the
<a href="https://teztnets.xyz/mondaynet-about">Mondaynet </a>testnet by providing
a <span class="caps">WASM</span> program — called a kernel — which interprets any Layer 2
operations targeting the rollup.</p>
<p>Aside from intra-rollup operations, these Layer 2 operations may also
transfer assets wrapped as tickets from Layer 1 accounts to rollups,
or produce asynchronous transfers of tickets from the rollup to Layer
1 contracts. A mechanism of data revelation is also introduced, so
that data from sources external to Tezos’ Layer 1 can be imported in a rollup.</p>
<p>There is no restriction on who can write and deploy a <span class="caps">WASM</span> kernel. An
example of a kernel facilitating transactions is available
<a href="http://www.gitlab.com/trili/kernel">here</a>, and we expect such kernels
to fully replace <a href="https://research-development.nomadic-labs.com/toru-introduction.html">Transaction Optimistic
Rollups</a>
once kernel-based rollups are activated on Mainnet.</p>
<p>We encourage the ecosystem to start identifying use cases for
kernel-based optimistic rollups, and to prepare for building <span class="caps">WASM</span>
kernels of their own. We are currently finalizing the necessary
documentation and guides, but anyone interested in getting started
with building kernels can reach out to
<a href="mailto:contact@nomadic-labs.com">contact@nomadic-labs.com</a>.</p>
<p>More info will be released soon. Stay tuned!</p>Kathmandu, the latest Tezos upgrade, is LIVE!2022-09-23T23:00:00+02:002022-09-23T23:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-09-23:/kathmandu-is-live.html<p>Tezos has activated its eleventh protocol upgrade, Kathmandu.</p><p>On September 23 2022 20:36:44 <span class="caps">UTC</span>, the Tezos blockchain successfully
upgraded by activating the <strong>Kathmandu</strong> proposal at block
<a href="https://tzkt.io/2736129/operations">#2,736,129</a>.</p>
<p>This 11th Tezos protocol upgrade was jointly developed by Nomadic
Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda, Functori,
Tweag, and <a href="https://gitlab.com/g.b.fefe">G.-B. Fefe</a> (an anonymous contributor).</p>
<p>Kathmandu’s main features are:</p>
<ul>
<li><strong><a href="https://research-development.nomadic-labs.com/next-generation-rollups.html">Smart contract optimistic
rollups:</a></strong>
the next generation of optimistic rollups for Tezos begin their
journey towards future integration in Tezos. At this stage, they are
available at the bleeding edge
<a href="https://teztnets.xyz/mondaynet-about">Mondaynet</a> and
<a href="https://teztnets.xyz/dailynet-about">Dailynet</a> test networks to
ensure the community has sufficient time to build integration,
tooling and applications.</li>
<li><strong><a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html#first-steps-towards-pipelined-block-validation">Pipelined validation of manager
operations:</a></strong>
increasing throughput for Tezos’ Mainnet, without compromising the
network’s safety. The contributions of this ongoing
<a href="https://gitlab.com/tezos/tezos/-/milestones/23#tab-issues">project</a>
to Kathmandu reduce the need to fully execute time-expensive
operations (like smart contract calls), before they reach a baker,
resulting in a faster propagation of new blocks and operations
across the network.</li>
<li><strong><a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html#improved-randomness-using-verifiable-delay-functions">Improved
randomness:</a></strong>
integration of Verifiable Delay Functions into the <a href="https://tezos.gitlab.io/kathmandu/randomness_generation.html">protocol’s
random seed
generation</a>,
reinforcing the security of the rights allocation mechanism.</li>
<li><strong><a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html#support-for-permanent-test-networks">Tailored governance support for permanent
testnets:</a></strong>
changes brought by Kathmandu will reduce the need for user-activated
upgrades in <a href="https://teztnets.xyz/ghostnet-about">Ghostnet</a>. The
Oxhead Alpha team will be able to centrally, and automatically,
upgrade this test network after a protocol proposal is elected <a href="https://tezos.gitlab.io/active/voting.html">in
the Promotion period</a> on
Tezos Mainnet.</li>
<li><strong><a href="https://www.marigold.dev/post/how-to-send-events-from-contract-fast">Event logging in Michelson smart
contracts:</a></strong>
this new feature will enable DApp developers to send publicly
visible on-chain messages in order to trigger effects in off-chain applications.</li>
<li>A <strong><a href="https://tezos.gitlab.io/protocols/014_kathmandu.html#increase-paid-storage">new operation for increasing paid
storage</a></strong>
of a smart contract.</li>
</ul>
<p>For more details, see our <a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html">Kathmandu preview
post</a>.</p>
<p>A deeper technical description can be found in the protocol proposal’s
<a href="https://tezos.gitlab.io/kathmandu/protocol.html">technical
documentation</a>, and a
complete list of changes is provided in Kathmandu’s
<a href="https://tezos.gitlab.io/protocols/014_kathmandu.html">changelog</a>.</p>
<h2 id="building-solid-foundations-for-the-future-now"><a class="toclink" href="#building-solid-foundations-for-the-future-now">Building solid foundations for the Future… now</a></h2>
<p>As the Tezos network evolves, and new exciting features boost
adoption, broad testing becomes increasingly important. Therefore, it
is paramount to have as many bakers, builders and users as possible in
our public <a href="https://tezos.gitlab.io/introduction/test_networks.html">test
networks</a>.</p>
<p>By running nodes, producing blocks, and deploying apps and
infrastructure as early as possible, the community can gain
significant foresight into the integration of new features. In
addition, protocol developers can learn valuable insights from
feedback, and adjust the design and implementation of features while
protocols are still being developed.</p>
<p>In particular, extensive testing of <a href="https://research-development.nomadic-labs.com/next-generation-rollups.html">next-generation optimistic
rollups</a>,
currently available on Mondaynet and Dailynet, is of paramount
importance, as these form the backbone of <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html">Tezos’ scaling
strategy</a>.</p>
<p>Read more about Tezos testnets <a href="https://teztnets.xyz/">here</a>, and
don’t hesitate to reach out in the <a href="tezos-dev.slack.com">Tezos Developer
Slack</a> or in the <a href="https://discord.com/invite/5v5QVKjvc2">Tezos
Discord</a> if you need help
getting started.</p>Monitoring Your Node with Octez Metrics2022-09-22T14:00:00+02:002022-09-22T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-09-22:/octez-metrics.html<p>Starting with Octez v14, the Tezos node fully integrates Octez Metrics, a powerful monitoring tool which scraps performance data out of your node and daemons and enables monitoring the overall Tezos network health. These metrics can be visualized using Grafazos, a collection of Tezos-specific Grafana dashboards. In this blog post, we’ll show you how to monitor your Tezos node — the <span class="caps">OG</span> way.
Starting with Octez v14, the Tezos node fully integrates Octez Metrics, a powerful monitoring tool which scraps performance data out of your node and daemons and enables monitoring the overall Tezos network health. These metrics can be visualized using Grafazos, a collection of Tezos-specific Grafana dashboards. In this blog post, we’ll show you how to monitor your Tezos node — the <span class="caps">OG</span> way.</p><!---
Monitoring the node
https://codimd.nomadic-labs.com/1QARmczzTZK67ctP-djSZg?view#Monitoring-the-node
---->
<h2 id="introduction"><a class="toclink" href="#introduction">Introduction</a></h2>
<p>Until now, the only tool developers had to monitor the behavior of their Tezos node was to look at the logs, <a href="https://tezos.gitlab.io/user/logging.html?highlight=log#node-specific-configuration">adjust the log verbosity</a>, and reconstruct all relevant information from this stream. But getting more insight into a node’s performance was tedious and difficult. For instance, the number of connected peers, the number of pending operations, or the number of times the validator switched branches, were not easy to observe continuously.</p>
<p>After a few iterations of different methods to gather node information and statistics that could be easily analyzed, we have recently chosen to include metrics within the node. With Octez Metrics it’s simple to get a myriad of statistics about your node — and quite efficiently so. You can also attach a <a href="https://grafana.com/">Grafana</a> dashboard to get a visual representation of how your node is performing. And with <a href="https://gitlab.com/nomadic-labs/grafazos">Grafazos</a>, you can get customized ready-to-use dashboards for monitoring your Tezos node.</p>
<p>A Grafazos dashboard looks like this:</p>
<figure align="center">
<img src="/images/octez-metrics-dashboard.gif" alt="Grafazos octez metrics dashboard" width="90%">
<figcaption align="center"> <b>Table 1:</b> Grafana dashboard of a Tezos node
</figcaption>
</figure>
<p>As you can immediately see at the top, the dashboard will tell you your node’s bootstrap status and whether it’s synchronized, followed by tables and graphs of other data points.</p>
<h2 id="node-metrics"><a class="toclink" href="#node-metrics">Node metrics</a></h2>
<p>In previous versions of Octez, a separate tool was needed for this task. <a href="https://gitlab.com/nomadic-labs/tezos-metrics">tezos-metrics</a> exported the metrics which were computed from the result of <span class="caps">RPC</span> calls to a running node. However, since the node <span class="caps">API</span> changed with each version, it required <code>tezos-metrics</code> to update alongside it, resulting in as many versions of tezos-metrics as Octez itself. Starting with <a href="https://tezos.gitlab.io/releases/version-14.html">Octez v14</a>, metrics were integrated into the node and can be exported directly, making it simple to set up.
Moreover, as the metrics are now generated by the node itself, no additional <span class="caps">RPC</span> calls are needed anymore. This is why the monitoring is now considerably more efficient!</p>
<h3 id="setting-up-octez-metrics"><a class="toclink" href="#setting-up-octez-metrics">Setting up Octez Metrics</a></h3>
<p>To use Octez Metrics, you just start your node with the metrics server enabled. The node integrates a server that registers the implemented metrics and outputs them for each <code>/metrics</code> <span class="caps">HTTP</span> request.</p>
<p>When you start your node you add the <code>--metrics-addr</code> option which takes as a parameter <code><ADDR:PORT></code> or <code><ADDR></code> or <code>:<PORT></code>. This option can be used either when starting your node, or in the configuration file (see <a href="https://tezos.gitlab.io/user/node-configuration.html">https://tezos.gitlab.io/user/node-configuration.html</a>).</p>
<p>Your node is now ready to have metrics scraped with requests to the metrics server. For instance, if the node server is configured to expose metrics on port 9932 (the default), then you can scrape the metrics with the request <code>http://localhost:9932/metrics</code>.
The result of the request is the list the node metrics described as:</p>
<div class="highlight"><pre><span></span><code>#HELP metric description
#TYPE metric type
octez_metric_name{label_name=label_value} x.x
</code></pre></div>
<p>Note the metrics are implemented to have the lowest possible impact on the node performance, and most of the metrics are only computed when scraping it. So starting the node with the metrics server shouldn’t be a cause for concern. More details on Octez Metrics can be found in the Tezos Developer Documentation: <a href="https://tezos.gitlab.io/user/node-monitoring.html">see here</a> for further detail on how to setup your monitoring; and <a href="https://tezos.gitlab.io/developer/openmetrics">here</a>, for the complete list of the metrics scrapped by the Octez node.</p>
<h3 id="types-of-metrics"><a class="toclink" href="#types-of-metrics">Types of metrics</a></h3>
<p>The available metrics give a full overview of your node, including its characteristics, status, and health. In addition, they can give insight into whether an issue is local to your node, or it is affecting the network at large — or both.</p>
<p>The metric <code>octez_version</code> delivers the node’s main properties through label-value pairs. It provides the node version or the network it is connected to.</p>
<p>Other primary metrics you likely want to see are the <em>chain validator</em><sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup> ones, which describe the status of your node: <code>octez_validator_chain_is_bootstrapped</code> and <code>octez_validator_chain_synchronisation_status</code>. A healthy node should always have these values set to 1. You can also see information about the <em>head</em> and <em>requests</em> from the chain validator.</p>
<p>There are two other validators, the <em>block validator</em><sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup> and the <em>peer validator</em><sup id="fnref:3"><a class="footnote-ref" href="#fn:3">3</a></sup>, which give you insight on how your node is handling the progression of the chain. You can learn more about the validators <a href="https://tezos.gitlab.io/shell/validation.html">here</a>.</p>
<p>To keep track of pending operations, you can check the <code>octez_mempool</code> metric.</p>
<p>You can get a view of your node’s connections with the <em>p2p layer</em> metrics (prefixed with <code>octez_p2p</code>). These metrics allows you to keep track of the <em>connections</em>, <em>peers</em> and <em>points</em> of your node.</p>
<p>The <a href="https://tezos.gitlab.io/shell/storage.html">store</a> can also be monitored with metrics on the save-point, checkpoint and caboose level, including the number of invalid blocks stored, the last written block size, and the last store merge time.</p>
<p>Finally, if you use the <span class="caps">RPC</span> server of your node, it is likely decisive in the operation of your node.
For each <span class="caps">RPC</span> called, two metrics are associated: <code>octez_rpc_calls_sum{endpoint="...";method="..."}</code> and <code>octez_rpc_calls_count{endpoint="...";method="..."}</code> (with appropriate label values). <em>call_sum</em> is the sum of the execution times, and <em>call_count</em> is the number of executions.</p>
<p><em>Note that the metrics described here are those available with Octez v14—it is likely to evolve with future Octez versions.</em></p>
<h3 id="dashboards"><a class="toclink" href="#dashboards">Dashboards</a></h3>
<p>While scraping metrics with server requests does give access to node metrics, it is, unfortunately, not enough for useful node monitoring. Since it only gives a single slice into the node’s health, you don’t really see what’s happening over time. Therefore, a more useful way of monitoring your node is to create a time series of metrics.</p>
<p>Indeed, if you liked the poster, why not see the whole movie?</p>
<p>The <a href="https://prometheus.io/">Prometheus</a> tool is designed for this purpose, to collect metric data over time.</p>
<p>In addition, in order to get the most out of your metrics, it should be associated with a visual dashboard.
A <a href="https://grafana.com/">Grafana</a> dashboard, generated by <a href="https://gitlab.com/nomadic-labs/grafazos">Grafazos</a> gives a greater view into your node. Once your node is launched, you can provide extracted time series of metrics to Grafana dashboards.</p>
<p>The Grafazos version for Octez v14 provides the following four ready-to-use dashboards:</p>
<ul>
<li><code>octez-compact</code>: A compact dashboard that gives a brief overview of the various node metrics on a single page.</li>
<li><code>octez-basic</code>: A basic dashboard with all the node metrics.</li>
<li><code>octez-with-logs</code>: Same as basic but also displays the node’s logs, with <a href="https://grafana.com/docs/loki/latest/clients/promtail/">Promtail</a> Promtail (for exporting the logs).</li>
<li><code>octez-full</code>: A full dashboard with the logs and hardware data. This dashboard should be used with <a href="https://www.netdata.cloud/">Netdata</a> (for supporting hardware data) in addition to Promtail.</li>
</ul>
<p>Note that the last two dashboards require the use of additional (though standard) tools for hardware metrics and logs (Netdata, Loki, and Promatail).</p>
<p>Let’s look at the <em>basic</em> dashboard in more detail. The dashboard is divided into several panels. The first one is the <em>node panel</em>, which can be considered the main part of the dashboard. This panel lays out the core information on the node such as its status, characteristics, and statistics on the node’s evolution (head level, validation time, operations, invalid blocks, etc.).</p>
<p>The others panels are specific to different parts of the node:</p>
<ul>
<li>the p2p layer;</li>
<li>the workers;</li>
<li>the <span class="caps">RPC</span> server;</li>
</ul>
<p>along with a miscellaneous section.</p>
<p>Some metrics are self-explanatory, such as <em><span class="caps">P2P</span> total connections</em>, which shows both the connections your node initiated and the number of connections initiated by peers. Another metric you may want to keep an eye on is <em>Invalid blocks history</em>, which should always be 0 — any other value would indicate something unusual or malicious is going on.</p>
<p>Another useful metric is the <em>Block validation time</em>, which measures the time between when a request is registered in the worker till the worker pops the request and marks it complete. This should generally be under 1 second. If it’s persistently longer, that could indicate trouble too.</p>
<figure align="center">
<img src="/images/metrics-block-validation-time.png" alt="Block validation time" width="90%">
<figcaption align="center"> <b>Graph 2:</b> Block validation time
</figcaption>
</figure>
<p>The <em><span class="caps">P2P</span> connections</em> graph will show you immediately if your node is having trouble connecting to peers, or if there’s a drop-off in the number of connections.
A healthy node should typically have a few dozen peer connections (depending on how it was configured).</p>
<figure align="center">
<img src="/images/metrics-p2p-connections.png" alt="P2P connections" width="90%">
<figcaption align="center"> <b>Graph 3:</b> <span class="caps">P2P</span> connections
</figcaption>
</figure>
<p>The <em>Peer validator</em> graph shows a number of different metrics including <em>unavailable protocols</em>. An up-to-date, healthy node should see this as a low number. If not it can indicate that your node is running an old version of Octez, or that your node is being fed bad data from peers.</p>
<p><em>Note again these dashboards are built for Octez v14 and are likely to evolve with the Octez versions</em>.</p>
<h3 id="working-with-grafazos"><a class="toclink" href="#working-with-grafazos">Working with Grafazos</a></h3>
<p>Grafazos allows you to set different options when generating the ready-to-use dashboards described above. For instance, you can specify the <code>node instance label</code>, which is useful for a dashboard that aims to monitor several nodes.</p>
<p>Furthermore, you can manually explore the metrics from the Prometheus data source with Grafana and design your own dashboards.
Or you can also use Grafazos to import ready-to-use dashboards for your node monitoring. You can find the packages stored <a href="https://gitlab.com/nomadic-labs/grafazos/-/packages">here</a>. There is a package for each version of Octez.</p>
<p>Grafana is a relatively user-friendly tool, so play with creating a custom dashboard as you like. You may also want to use the “explore” section of Grafana. Grafazos is also particularly useful in automatic deployment of Tezos nodes via provisioning tools such as Puppet or Ansible.</p>
<h3 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h3>
<p>We developed Octez Metrics to give Tezos users a better insight into how their node is performing, and to observe the overall network health. This is a product that will continuously improve, and we encourage node operators and bakers to suggest new features and additional metrics that they would like to see. We recognize that the best way to keep your node healthy — and in turn, to keep the entire Tezos network healthy — is to provide everyone the tools needed to monitor their setup.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>The chain validator is responsible for handling valid blocks and selecting the best head for its chain. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>The block validator validates blocks and notifies the corresponding chain validator. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>Each peer validator treats new head proposals from its associated peer, retrieving all the operations, and if valid, triggers a validation of the new head. <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>Verify, but test: extracting QCheck Property-Based Tests from F* specifications2022-09-12T17:00:00+02:002022-09-12T17:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-09-12:/verify-but-test.html<p>The formal, mechanized, verification of computer programs is a particularly difficult task in fast-evolving and complex development environments, as it typically is a slow process. To speed things up, verification efforts often target abstract models of these programs, and many implementation details are often left unverified in critical code. In this blog post, we propose to bridge the verification gap between a mechanized model and its implementation by leveraging property-based testing. Concretely, we present a light-weight automated tool for extracting F* specifications as OCaml predicates for their execution as a QCheck tests.</p><p>The focus of this blog post is on the automated extraction of an F* specification to OCaml and its execution as a QCheck Property-Based Test against an OCaml implementation.
This work was done by Antonio Locascio during a 4 month <a href="https://research-development.nomadic-labs.com/three-questions-to-nomadic-labs-interns-antonio-locascio.html">internship at Nomadic Labs</a> under the supervision of Germán Delbianco and Marco Stronati.</p>
<p>This work will also be presented at the <a href="https://icfp22.sigplan.org/details/mlfamilyworkshop-2022-papers/11/Verify-but-test-extracting-property-based-tests-from-F-specifications"><span class="caps">ML</span> 2022 workshop</a>.</p>
<h2 id="introduction"><a class="toclink" href="#introduction">Introduction</a></h2>
<p>A typical workflow when developing high-assurance software is to start with a prototype to better understand the problem at hand, write a specification for the desired system (having learnt from all the mistakes in the process of implementing said prototype), write a reference implementation of the specification, and then formally verify that the implementation matches the specification, a process that is slow and costly.</p>
<p>However in a fast paced development cycle, the not-yet-verified implementation is often sent to production, continues to be improved over time and becomes a moving target for the verification effort, which often does not bear fruit. This is not a new problem. One of the main goals behind the development of novel formal verification tools has been to reduce the cost of verification, to allow it to be better integrated in the development cycle (see for example <a href="https://www.fstar-lang.org/tutorial/book/index.html">proof oriented programming</a>).</p>
<p>In many cases it is possible today to extract an implementation that matches a specification, thus merging the last two steps of the workflow described above.</p>
<p>Going a step further, we would like to start sketching a specification while we develop the prototype and use this model to test the prototype itself. Thus moving from a linear development process to an iterative one, where implementation and specification are constantly refined and adapted. While abstracting away implementation details in the specification, the <em>verification gap</em> problem arises: given that the verified model is loosely coupled with the implementation, do the properties that we have proved for the model really hold for the implementation?</p>
<p>One way of addressing the <em>verification gap</em> problem is to rely on Property-Based Testing (<span class="caps">PBT</span>), that is, to extract the model as properties that can be executed as tests on the implementation. The focus of this work is on a lightweight automated tool for extracting F* specifications to OCaml, and its execution as a QCheck test against an OCaml implementation. We propose to OCaml programmers to write specifications as F* programs, instead of manually writing QCheck tests. By doing this, our tool will automatically extract the specification as QCheck properties, which can then be used to test either the code extracted from the model, or another implementation.</p>
<p>Thanks to this approach, rather than waiting for a finished implementation and working on a complete formal verification, we can move along a <em>verification spectrum</em>. Start with a prototype implementation tested by <span class="caps">PBT</span> on its co-developed abstract model, refine the model over time in order to extract parts of it and replace them in the implementation until (possibly) a full extraction can be achieved.</p>
<p>It should be noted that a full extraction is sometimes too costly, as it may require the model to precisely represent complex parts of the code that are not essential to the program’s core functionality. However, being able to find the sweet spot along this spectrum where only the crucial parts have been verified is still very valuable and common practice.</p>
<p>Even where a full extraction can be achieved, the complex toolchains and large trusted computing base can still benefit from a testing infrastructure.
For example it is common practice to assume (and axiomatize) the existence of a <code>List</code> library that, during extraction, will be linked to the <code>List</code> module of the standard library, despite the fact that there is no formal connection between the two.
For this reason, even if our properties have already been proved in the model, they should also be tested in the extracted implementation.
In other words the <em>verification gap</em> may never be really closed in full, even at the end of the spectrum, and property based testing is a way to account for it in the development process.</p>
<p>The rest of the document is structured as follows: we first give a brief overview of the F* language and Property-Based Testing. Then, we give a detailed account of our proposed specification extraction tool for F*. After that, we present the case study for our workflow, Incremental Merkle Trees, which illustrates the wide spectrum of verification and testing strategies. Next, we describe our experience of working with F*, and explain our contributions to that project. Finally, we discuss related projects and some future work.</p>
<h2 id="overview"><a class="toclink" href="#overview">Overview</a></h2>
<p>The workflow presented in this post, and the tooling developed, requires choosing a verification <em>framework</em> capable of extracting correct code to the target programming language, and a <span class="caps">PBT</span> library for the target language. In particular, given our OCaml codebase, we choose <a href="https://www.fstar-lang.org/">F*</a> as a mechanization medium, and <a href="https://c-cube.github.io/qcheck/">QCheck</a> as a testing library. In the following, we give a brief overview of both systems.</p>
<h3 id="an-f-primer"><a class="toclink" href="#an-f-primer">An F* primer</a></h3>
<p><a href="https://www.fstar-lang.org/">F*</a> is an <span class="caps">ML</span>-style functional programming language aimed at formal verification. F* combines automated verification through an <span class="caps">SMT</span> solver with the interactive functionality of a proof assistant based on dependent types, making it a hybrid verification tool. After the verification step, an F* program can be extracted to OCaml in order to be executed.
Additionally F* supports meta-programming, which is mainly developed to support proving by tactics, but is a crucial feature for this work as it allows the extraction of specifications.</p>
<p>We’ll work through an example to introduce F*’s fundamental aspects. Let’s take a look at the following function, which reverses a list:</p>
<div class="highlight"><pre><span></span><code><span class="k">val</span><span class="w"> </span><span class="n">reverse</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="kt">list</span><span class="w"> </span><span class="n">α</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Tot</span><span class="w"> </span><span class="o">(</span><span class="kt">list</span><span class="w"> </span><span class="n">α</span><span class="o">)</span>
<span class="k">let</span><span class="w"> </span><span class="nv">rec</span><span class="w"> </span><span class="n">reverse</span><span class="w"> </span><span class="n">l</span><span class="w"> </span><span class="o">=</span>
<span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">l</span><span class="w"> </span><span class="k">with</span>
<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="bp">[]</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="bp">[]</span>
<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">::</span><span class="w"> </span><span class="n">xs</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">reverse</span><span class="w"> </span><span class="n">xs</span><span class="w"> </span><span class="o">@</span><span class="w"> </span><span class="o">[</span><span class="n">x</span><span class="o">]</span>
</code></pre></div>
<p>Those familiar with <span class="caps">ML</span>-style languages (such as OCaml) will straight away feel at home with F*’s syntax.
One difference with OCaml’s syntax is that in F* signatures are usually given before a definition.
These are a central aspect of F*, because of its rich type system.</p>
<p>One key feature of the type system is the effect system. Notice that in the return type of the example, <code>Tot</code> is an <em>effect</em> annotation, expressing that the function is total, pure and non-diverging. The generic shape of a function type in F* is <code>α -> E β</code>, where <code>E</code> is an effect. Additional built-in effects are: <code>Dv</code>, for diverging functions, <code>ST</code> for stateful computations, and <code>ML</code> for functions with arbitrary side-effects. <code>Tot</code> is the default effect, meaning that its declaration can be omitted.</p>
<p>Refinement types are another important part of F*’s type system. The refinement type <code>x:T{φ(x)}</code> is a <a href="https://en.wikipedia.org/wiki/Subtyping">sub-type</a> of <code>T</code> for which all its inhabitants satisfy the property <code>φ</code>, also called the <em>refinement</em>. In other words, type <code>x:T{φ(x)}</code> is the type of values <code>x</code> from <code>T</code> for which <code>φ(x)</code> is true. For simplicity, we can assume that the predicates <code>φ</code> are total functions returning a boolean, although they can be more <a href="http://fstar-lang.org/tutorial/tutorial.html#sec-bool-versus-type">general</a>. For instance, we can use them to give a more precise signature to our example:</p>
<div class="highlight"><pre><span></span><code><span class="k">val</span><span class="w"> </span><span class="n">reverse</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">l</span><span class="o">:</span><span class="kt">list</span><span class="w"> </span><span class="n">α</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Tot</span><span class="w"> </span><span class="o">(</span><span class="n">l'</span><span class="o">:(</span><span class="kt">list</span><span class="w"> </span><span class="n">α</span><span class="o">){</span><span class="n">length</span><span class="w"> </span><span class="n">l'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">length</span><span class="w"> </span><span class="n">l</span><span class="o">})</span>
</code></pre></div>
<p>Here, we use the refinement type to state that we expect the resulting list to have the same length as the original one. To refer to the original list, we needed to name the argument (<code>l</code>) in the signature, making this a dependent function type.</p>
<p>F* provides an alternative syntax for writing a function’s pre- and post-conditions explicitly. Let’s look at it with a new example:</p>
<div class="highlight"><pre><span></span><code><span class="k">val</span><span class="w"> </span><span class="n">tail</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">l</span><span class="o">:</span><span class="kt">list</span><span class="w"> </span><span class="n">α</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Pure</span><span class="w"> </span><span class="o">(</span><span class="kt">list</span><span class="w"> </span><span class="n">α</span><span class="o">)</span>
<span class="w"> </span><span class="o">(</span><span class="n">requires</span><span class="w"> </span><span class="o">(</span><span class="n">length</span><span class="w"> </span><span class="n">l</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="o">))</span>
<span class="w"> </span><span class="o">(</span><span class="n">ensures</span><span class="w"> </span><span class="o">(</span><span class="k">fun</span><span class="w"> </span><span class="n">l'</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">length</span><span class="w"> </span><span class="n">l'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">length</span><span class="w"> </span><span class="n">l</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="o">))</span>
</code></pre></div>
<p><code>Pure</code> is just a version of the <code>Tot</code> effect that, in addition to the type of the return value (<code>list α</code>), is parameterized by a pre-condition (<code>requires</code> clause) and a post-condition (<code>ensures</code> clause). Then, this signature is equivalent to:</p>
<div class="highlight"><pre><span></span><code><span class="k">val</span><span class="w"> </span><span class="n">tail</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">l</span><span class="o">:(</span><span class="kt">list</span><span class="w"> </span><span class="n">α</span><span class="o">){</span><span class="n">length</span><span class="w"> </span><span class="n">l</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">0</span><span class="o">}</span><span class="w"> </span><span class="o">-></span>
<span class="w"> </span><span class="n">Tot</span><span class="w"> </span><span class="o">(</span><span class="n">l'</span><span class="o">:(</span><span class="kt">list</span><span class="w"> </span><span class="n">α</span><span class="o">){</span><span class="n">length</span><span class="w"> </span><span class="n">l'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">length</span><span class="w"> </span><span class="n">l</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="o">})</span>
</code></pre></div>
<p>It’s not always desirable to pollute a function’s signature with properties about it. That’s when <em>lemmas</em> come into play. In F*, a lemma is a computationally irrelevant function (a total function returning <code>unit</code>), used to state some property about a program.
For example, we could state (and subsequently prove) that our <code>reverse</code> function is <em>involutive</em> with the following lemma:</p>
<div class="highlight"><pre><span></span><code><span class="k">val</span><span class="w"> </span><span class="n">reverse_involutive</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">l</span><span class="o">:</span><span class="kt">list</span><span class="w"> </span><span class="n">α</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Lemma</span><span class="w"> </span><span class="o">(</span><span class="n">l</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">reverse</span><span class="w"> </span><span class="o">(</span><span class="n">reverse</span><span class="w"> </span><span class="n">l</span><span class="o">))</span>
</code></pre></div>
<p>A more detailed introduction to the F* language can be found in its <a href="https://www.fstar-lang.org/tutorial/">tutorial</a>.</p>
<h3 id="property-based-testing-with-qcheck"><a class="toclink" href="#property-based-testing-with-qcheck">Property-Based Testing with QCheck</a></h3>
<p>Property-based testing (or <span class="caps">PBT</span> for short) is a testing discipline in which properties about programs are checked to be true by validating them on a large number of randomly generated examples.
Property-based testing a program generally requires the user to do two
things: (i) to define the desired properties about the code, and (ii)
to provide functions for generating random inputs for those properties
(commonly known as <em>generators</em>).</p>
<p>To aid the definition of these properties and custom data type generators, several libraries have been developed. One of these is <a href="https://c-cube.github.io/qcheck/">QCheck</a>, which offers a wide range of combinators to do <span class="caps">PBT</span> in OCaml. In this project we present a tool that automatically extracts QCheck properties from F* specifications. The generator definition, however, is still left to the user, as explained in the following section.</p>
<h2 id="specification-extraction-toolchain"><a class="toclink" href="#specification-extraction-toolchain">Specification Extraction Toolchain</a></h2>
<p>Let’s start with some motivation for using a specification extraction tool. Say you have an OCaml program, or even an idea for one, and you want to write a specification for it. We propose that, instead of directly writing QCheck tests for it, you write the specification as an F* program. If you do this, our tool will automatically extract the specification as QCheck properties, which can be used to test both the code extracted from the model and the original implementation.</p>
<p>It’s not required to do a full verification of the code in order to use our tool. As previously mentioned, one could specify an abstract model of the implementation, whose properties could be tested against the more complex OCaml implementation. Further, one might even write the specification as an axiomatization of the implementation, i.e assuming the properties and leaving their actual proofs as a future step.</p>
<p>In order to extract a specification written in F* as properties for property-based testing we introduce a <a href="https://arxiv.org/abs/1803.06547">Meta-F*</a> tactic called <code>extract_spec</code>, which is defined in the <code>SpecExtraction</code> module.</p>
<p>This tactic does two things. First, it extracts a function’s pre- and post-conditions as OCaml boolean functions. Second, it synthesizes the QCheck boilerplate code for defining the test.</p>
<h3 id="functionality"><a class="toclink" href="#functionality">Functionality</a></h3>
<p>Before getting into the tool’s design, let’s take a look at a simple example of its usage. Its main functionality consists of extracting a function’s or lemma’s pre- and post-conditions as OCaml boolean functions. For instance, given the following F* function:</p>
<div class="highlight"><pre><span></span><code><span class="k">val</span><span class="w"> </span><span class="n">foo</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">x</span><span class="o">:</span><span class="n">T</span><span class="o">{</span><span class="n">P1</span><span class="w"> </span><span class="n">x</span><span class="o">}</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Pure</span><span class="w"> </span><span class="n">T'</span><span class="w"> </span><span class="o">(</span><span class="n">requires</span><span class="w"> </span><span class="o">(</span><span class="n">P2</span><span class="w"> </span><span class="n">x</span><span class="o">))</span><span class="w"> </span><span class="o">(</span><span class="n">ensures</span><span class="w"> </span><span class="o">(</span><span class="k">fun</span><span class="w"> </span><span class="n">y</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Q</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="n">y</span><span class="o">))</span>
<span class="k">let</span><span class="w"> </span><span class="nv">foo</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">...</span>
</code></pre></div>
<p>One can call the <code>extract_spec</code> tactic through a <em>splice</em>, which inserts syntax generated through Meta-F*. This is done by adding the following line (anywhere <code>foo</code> is in scope):</p>
<div class="highlight"><pre><span></span><code><span class="o">%</span><span class="n">splice</span><span class="bp">[]</span><span class="w"> </span><span class="o">(</span><span class="n">extract_spec</span><span class="w"> </span><span class="o">(`%</span><span class="n">foo</span><span class="o">))</span>
</code></pre></div>
<p>As a result, in the extracted OCaml code of the module where the splice was called, the following predicates will be defined:</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="n">foo_pre</span> <span class="o">=</span> <span class="k">fun</span> <span class="n">x</span> <span class="o">-></span> <span class="nc">P1</span> <span class="n">x</span> <span class="o">&&</span> <span class="nc">P2</span> <span class="n">x</span>
<span class="k">let</span> <span class="n">foo_post</span> <span class="o">=</span> <span class="k">fun</span> <span class="n">x</span> <span class="o">-></span> <span class="k">fun</span> <span class="n">y</span> <span class="o">-></span> <span class="nc">Q</span> <span class="n">x</span> <span class="n">y</span>
</code></pre></div>
<p>As shown in this example, a function’s pre-condition is parametrized by the function’s arguments, while its post-condition takes as an additional argument the function’s return value.</p>
<p>The second step, generating the QCheck tests, is performed by the <code>make_test</code> tactic (called by <code>extract_spec</code>). Continuing with the example, this tactic generates the following declarations:</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="o">(</span><span class="n">gen_foo_args</span> <span class="o">:</span> <span class="nc">T</span> <span class="nn">FStar_QCheck</span><span class="p">.</span><span class="n">arbitrary</span><span class="o">)</span>
<span class="o">=</span> <span class="nn">FStar_QCheck</span><span class="p">.</span><span class="n">undefined_gen</span> <span class="s2">"Generator for foo's input not yet implemented"</span>
<span class="k">let</span> <span class="o">(</span><span class="n">test_foo_spec</span> <span class="o">:</span> <span class="nn">FStar_QCheck</span><span class="p">.</span><span class="n">test_t</span><span class="o">)</span> <span class="o">=</span>
<span class="nn">FStar_QCheck</span><span class="p">.</span><span class="n">test_make</span> <span class="s2">"foo_spec"</span> <span class="n">gen_foo_args</span>
<span class="o">(</span><span class="k">fun</span> <span class="n">a0</span> <span class="o">-></span>
<span class="k">match</span> <span class="n">a0</span> <span class="k">with</span>
<span class="o">|</span> <span class="n">x</span> <span class="o">-></span> <span class="o">(</span><span class="nn">FStar_QCheck</span><span class="p">.</span><span class="n">assume_</span> <span class="o">(</span><span class="n">foo_pre</span> <span class="n">x</span><span class="o">);</span>
<span class="n">foo_post</span> <span class="n">x</span> <span class="o">(</span><span class="n">foo</span> <span class="n">x</span><span class="o">)))</span>
</code></pre></div>
<p>The first of these corresponds to a template definition of the QCheck generator for <code>foo</code><span class="quo">‘</span>s arguments. Deriving the generators is outside the scope of this project. Some discussion regarding this can be found in this <a href="https://gitlab.com/nomadic-labs/sapling-verification/-/issues/20">issue</a>.</p>
<p>The second is the definition of the QCheck test, in which the pre-condition is assumed and the post-condition is checked. The <code>FStar_QCheck</code> module is just a simple interface wrapper to expose some <code>QCheck</code> functions in F*.</p>
<h3 id="tools-design"><a class="toclink" href="#tools-design">Tool’s design</a></h3>
<p>The tool’s design revolves around Meta-F* tactics. Meta-F* is a metaprogramming framework for F*, i.e. a library for manipulating F* terms from within F*. A <em>tactic</em>, then, is simply a function that uses Meta-F*.</p>
<p>The bulk of the specification extraction work is performed by the <code>extract_spec</code> tactic, which takes as argument the name of the function the user wants to extract a specification from. We’ll give an overlook of the algorithm it implements:</p>
<ol>
<li>The first step is to query the environment to get the function’s type. This could either be annotated in a signature or inferred by the typechecker.</li>
<li>The retrieved type is subject to some (mostly trivial) transformations. The most interesting preprocessing step is <em>synonym resolution</em> (for example turning <code>nat</code> into <code>v:int{v >= 0}</code>), which is crucial to capture properties hidden in refinements.</li>
<li>Next, the type is traversed, one arrow at a time, collecting the names, types and refinements for each argument. For instance, for a type <code>x1:T1{P1 x1} -> x2:T2{P2 x1 x2} -> C</code>, <code>collect_args</code> will compute the list: <code>[(x1;T1; fun x1 -> P1 x1); (x2;T2; fun x1 x2 -> P2 x1 x2)]</code>. For each refinement we abstract all the previous binders. For now, we leave the final computation <code>C</code> untouched. It’s important to note that an argument’s type might have nested refinements, e.g. <code>x:(v:int{v >= 0}){x < 5}</code>. In this step, these refinements are flattened to get <code>fun x -> x >= 0 && x < 5</code>.</li>
<li>All of the arguments’ refinements are then joined to get the function’s implicit pre-condition.</li>
<li>Finally, the final computation type <code>C</code> is inspected. From it the post-condition (both from an ensures clause and the return type’s refinement) and potential explicit pre-condition (requires clause) are retrieved. When there is an explicit pre-condition, it is joined with the computed implicit pre-condition. There’s special treatment for lemmas, as in that case the post-condition doesn’t require an extra argument.</li>
</ol>
<p>Most of this procedure is performed by the <code>pre_post</code> tactic defined in <code>SpecExtraction.PrePost</code>. The QCheck test boilerplate generation, defined in <code>SpecExtraction.TestGeneration.make_test</code>, is not discussed, as it’s mostly trivial syntax declaration.</p>
<h3 id="limitations"><a class="toclink" href="#limitations">Limitations</a></h3>
<p>Our specification extraction tool has two limitations worth mentioning.</p>
<p>The first one is that only <code>bool</code> refinements are supported. This is necessary, as <code>Type</code> predicates might not be computable, for they can include universal and existential quantifiers anywhere in the term.</p>
<p>The second is that only pure functions work with the specification extraction mechanism for now. Although some other effect monads might be easily supported, this is not true in general. Supporting the ever-so-present <code>ST</code> effect, that carries a model of the heap in the type, for instance, doesn’t seem a trivial matter.</p>
<h2 id="case-study-incremental-merkle-trees"><a class="toclink" href="#case-study-incremental-merkle-trees">Case study: Incremental Merkle Trees</a></h2>
<p>We applied our proposed workflow of verification and testing to the implementation of the <em>Incremental Merkle Tree</em> data structure, used in the <a href="https://tezos.gitlab.io/alpha/sapling.html"><em>Sapling protocol</em></a>. This was a suitable target for our project, as:</p>
<ul>
<li>It’s a self-contained piece of code, whose only important dependencies are well-specified cryptographic primitives.</li>
<li>The data structure has clear invariants, the preservation of which is not trivial.</li>
<li>There are multiple implementations for it, of varying complexity, which help showcase the wide spectrum of verification and testing models.</li>
</ul>
<p>We now proceed to give a brief introduction to Sapling and IMTs, before delving into a detailed account of our case study.</p>
<h3 id="sapling"><a class="toclink" href="#sapling">Sapling</a></h3>
<p>Sapling is a protocol used in Tezos that enables privacy-preserving transactions of tokens. For storage purposes, this protocol uses the <em>Incremental Merkle Tree</em> data structure, or <span class="caps">IMT</span>. This <span class="caps">IMT</span> structure is simply a fixed height Merkle tree, in which the leaves are only stored on the last level in the leftmost positions and cannot be deleted. Because of their use in Zero-Knowledge proofs, the IMTs must always be considered to be full trees of fixed capacity. In the next <a href="#incremental-merkle-trees">section</a>, we provide a more detailed description of IMTs. For more documentation, refer to the protocol’s <a href="https://github.com/zcash/zips/blob/2e26bb072dfd5f842fe9e779bdec8cabeb4fa9bf/protocol/protocol.pdf">specification</a>.</p>
<p>In Tezos, there are currently two implementations of Sapling. The first one, found in <code>lib_sapling</code>, implements IMTs through purely functional ADTs. In contrast, the second implementation, which is part of the economic protocol, is written in a stateful style to make use of the protocol’s key-value storage. While the latter is more efficient, the former is simpler and thus easier to verify.</p>
<h3 id="incremental-merkle-trees"><a class="toclink" href="#incremental-merkle-trees">Incremental Merkle Trees</a></h3>
<p>An Incremental Merkle Tree of height <code>h</code> contains <code>2^h</code> leaves and <code>h + 1</code> levels of nodes, with all leaves at level <code>0</code> and root at level <code>h</code>.</p>
<p>For now, we focus on the <span class="caps">IMT</span> implementation found in the <a href="https://gitlab.com/tezos/tezos/-/blob/master/src/lib_sapling/storage.ml"><code>Storage</code> module</a> from <code>lib_sapling</code>. This implementation includes the following definition of an algebraic data type <code>tree</code> to represent IMTs:</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="n">tree</span> <span class="o">=</span> <span class="nc">Empty</span> <span class="o">|</span> <span class="nc">Leaf</span> <span class="k">of</span> <span class="nn">C</span><span class="p">.</span><span class="nn">Commitment</span><span class="p">.</span><span class="n">t</span> <span class="o">|</span> <span class="nc">Node</span> <span class="k">of</span> <span class="o">(</span><span class="nn">H</span><span class="p">.</span><span class="n">t</span> <span class="o">*</span> <span class="n">tree</span> <span class="o">*</span> <span class="n">tree</span><span class="o">)</span>
</code></pre></div>
<p>This type of trees have <em>commitments</em> in their leaves and <em>hashes</em> (<code>H.t</code>) in their nodes. Although both of these have the same internal representation, they are differentiated because the commitments correspond to the values stored in the tree, while nodes’ hashes are used to preserve integrity (as in any Merkle tree).</p>
<p>The trees are always treated as being full, using the default value <code>H.uncommitted 0</code> for unused leaves. All the nodes at the same level of an empty tree have the same hash, which can be computed from the default value of the leaves. These hashes are stored in the <code>H.uncommitted</code> list, so <code>H.uncommitted n</code> is the hash of an empty tree of height <code>n</code>.</p>
<p>To avoid storing huge empty trees, any subtree filled with default values is represented by the <code>Empty</code> constructor and given its height it’s possible to compute its hash using the <code>H.uncommitted</code> list. Because of this, we’ll sometimes refer to this representation of IMTs as <em>compressed trees</em>.</p>
<p>The leaves are indexed by their position <code>pos</code>, ranging from <code>0</code> to <code>(2^h) - 1</code>. The tree is incremental in the sense that leaves cannot be modified but only added and exclusively in successive positions from the leftmost to the rightmost leaf — i.e., from <code>0</code> to <code>(2^h) - 1</code>.</p>
<h3 id="imt-f-model"><a class="toclink" href="#imt-f-model"><span class="caps">IMT</span> F* Model</a></h3>
<h4 id="commitments-axiomatization"><a class="toclink" href="#commitments-axiomatization">Commitments axiomatization</a></h4>
<p>Let’s begin our dive into the model by focusing on the <code>Commitments</code> module. This interface axiomatizes the cryptographic primitives needed for IMTs, namely commitments and hashes. Here is where most of the model’s assumptions are introduced, most notably the following lemma:</p>
<div class="highlight"><pre><span></span><code><span class="k">val</span><span class="w"> </span><span class="n">perfect_merkle</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">h</span><span class="o">:</span><span class="n">v_height</span><span class="w"> </span><span class="o">-></span>
<span class="w"> </span><span class="n">Lemma</span><span class="w"> </span><span class="o">(</span><span class="n">ensures</span><span class="w"> </span><span class="o">(</span><span class="n">forall</span><span class="w"> </span><span class="n">h1</span><span class="w"> </span><span class="n">h2</span><span class="w"> </span><span class="n">h3</span><span class="w"> </span><span class="n">h4</span><span class="o">.</span>
<span class="w"> </span><span class="n">merkle_hash</span><span class="w"> </span><span class="n">h</span><span class="w"> </span><span class="n">h1</span><span class="w"> </span><span class="n">h2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">merkle_hash</span><span class="w"> </span><span class="n">h</span><span class="w"> </span><span class="n">h3</span><span class="w"> </span><span class="n">h4</span>
<span class="w"> </span><span class="o">==></span><span class="w"> </span><span class="n">h1</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">h3</span><span class="w"> </span><span class="o">/</span><span class="err">\</span><span class="w"> </span><span class="n">h2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">h4</span><span class="o">))</span>
</code></pre></div>
<p>which states that the assumed Merkle hash function is injective at every valid level (<code>h</code>). This property, which can be read as “if two outputs are equal, then their inputs must be equal”, models the absence of collisions in an ideal hash function.</p>
<p>This module interface’s OCaml realization (<code>ml/Commitments.ml</code>) is just a wrapper for the corresponding functions in <code>lib_sapling</code>.</p>
<h4 id="machine-integers"><a class="toclink" href="#machine-integers">Machine integers</a></h4>
<p>The second aspect of the model we’ll discuss is the use of machine integers. Our initial version of the model was based on F* unbounded <code>int</code>s, which are implemented as <a href="https://github.com/ocaml/Zarith">Zarith ints</a>. In order to make the model as close to the implementation as possible, we refined the specification using machine integer types, following the ones used in the OCaml code. This not only means that we prove the absence of arithmetic over/underflow in our model, but also that the extracted code should have the same performance (w.r.t. arithmetic operations) as the original implementation.</p>
<p>In the OCaml implementation from <code>lib_sapling</code>, both <code>Stdint</code> and OCaml native ints are used. For the former, F* already provides <code>[U]IntN</code> modules in the standard library, that are extracted to their <code>Stdint</code> counterpart.</p>
<p>Things are a little trickier for OCaml’s native <code>int</code> type, whose representation is architecture dependent. For this model, we assume a 64 bit architecture, so we need to deal with 63 bit integers. To do so, we provide an interface module <code>Int63</code> following the structure of those provided by the standard library. We also include an OCaml realization for it (<code>ml/Int63.ml</code>), that implements them using the native <code>int</code> type.</p>
<p>Refining the specification using these representations of machine integers proved to be much smoother than what we initially feared. There was a downside, however, from an extraction point of view, which will be discussed in a future <a href="#extraction-of-machine-integers">section</a>.</p>
<h4 id="main-tree-model"><a class="toclink" href="#main-tree-model">Main Tree model</a></h4>
<p>Now we can delve into the main part of the model for IMTs, defined in the <code>Tree</code> module. There are three important components that build up this model, each implemented as a sub-module. The first of these is <code>Tree.Data</code>, which introduces the underlying data type that we use to represent IMTs.</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span><span class="w"> </span><span class="nc">tree</span><span class="w"> </span><span class="n">α</span><span class="w"> </span><span class="n">β</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">Type</span><span class="w"> </span><span class="o">=</span>
<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Empty</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">tree</span><span class="w"> </span><span class="n">α</span><span class="w"> </span><span class="n">β</span>
<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Leaf</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">α</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">tree</span><span class="w"> </span><span class="n">α</span><span class="w"> </span><span class="n">β</span>
<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Node</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">β</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">l</span><span class="o">:</span><span class="n">tree</span><span class="w"> </span><span class="n">α</span><span class="w"> </span><span class="n">β</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">r</span><span class="o">:</span><span class="n">tree</span><span class="w"> </span><span class="n">α</span><span class="w"> </span><span class="n">β</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">tree</span><span class="w"> </span><span class="n">α</span><span class="w"> </span><span class="n">β</span>
</code></pre></div>
<p>As implied by its name, the type <code>tree</code> models <code>lib_sapling</code><span class="quo">‘</span>s type of the same name. The only difference is that in our model, the types of the values of the leaves and internal nodes are parameterized. For our <span class="caps">IMT</span> spec, <code>α</code> will generally be instantiated with the type for commitments and <code>β</code> with the one for hashes.</p>
<p>This <code>tree</code> type, then, is a direct translation of its OCaml counterpart. Things only start getting interesting when the type is refined with richer properties. Hence, in <code>Tree.Properties</code> the core properties used to specify IMTs are presented, alongside some lemmas about them. Let’s take a look at the two properties that give IMTs their name.</p>
<div class="highlight"><pre><span></span><code><span class="k">val</span><span class="w"> </span><span class="n">incremental</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">h</span><span class="o">:</span><span class="nn">C</span><span class="p">.</span><span class="n">v_height</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">t</span><span class="o">:</span><span class="n">tree</span><span class="w"> </span><span class="n">α</span><span class="w"> </span><span class="n">β</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Tot</span><span class="w"> </span><span class="kt">bool</span>
<span class="k">let</span><span class="w"> </span><span class="nv">incremental</span><span class="w"> </span><span class="n">h</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">=</span>
<span class="w"> </span><span class="n">valid</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="n">left_leaning</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="n">balanced</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="n">has_height</span><span class="w"> </span><span class="n">h</span><span class="w"> </span><span class="n">t</span>
</code></pre></div>
<p>The first one is incrementality. This property essentially states that the leaves are filled in consecutive positions from left to right. Additionally, some other structural invariants are included in this property, such as the fact that no internal node should have two empty sub-trees. The next one, <code>merkle</code>, checks that the values in the internal nodes actually are the hashes of their sub-trees.</p>
<div class="highlight"><pre><span></span><code><span class="k">val</span><span class="w"> </span><span class="n">merkle</span><span class="w"> </span><span class="o">:</span>
<span class="w"> </span><span class="n">h</span><span class="o">:</span><span class="nn">C</span><span class="p">.</span><span class="n">v_height</span><span class="w"> </span><span class="o">-></span>
<span class="w"> </span><span class="n">t</span><span class="o">:</span><span class="n">tree</span><span class="w"> </span><span class="nn">C</span><span class="p">.</span><span class="n">t_C</span><span class="w"> </span><span class="nn">C</span><span class="p">.</span><span class="n">t_H</span><span class="o">{</span><span class="n">valid</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">/</span><span class="err">\</span><span class="w"> </span><span class="n">has_height</span><span class="w"> </span><span class="n">h</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">/</span><span class="err">\</span><span class="w"> </span><span class="n">balanced</span><span class="w"> </span><span class="n">t</span><span class="o">}</span><span class="w"> </span><span class="o">-></span>
<span class="w"> </span><span class="n">Tot</span><span class="w"> </span><span class="kt">bool</span>
<span class="w"> </span><span class="o">(</span><span class="n">decreases</span><span class="w"> </span><span class="o">%[</span><span class="n">v63</span><span class="w"> </span><span class="n">h</span><span class="o">])</span>
<span class="k">let</span><span class="w"> </span><span class="nv">rec</span><span class="w"> </span><span class="n">merkle</span><span class="w"> </span><span class="n">h</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">=</span>
<span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="k">with</span>
<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Empty</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="k">true</span>
<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Leaf</span><span class="w"> </span><span class="o">_</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="k">true</span>
<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">Node</span><span class="w"> </span><span class="n">ha</span><span class="w"> </span><span class="n">l</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">-></span>
<span class="w"> </span><span class="k">let</span><span class="w"> </span><span class="nv">hs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">pred63</span><span class="w"> </span><span class="n">h</span><span class="w"> </span><span class="k">in</span>
<span class="w"> </span><span class="n">ha</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">hash</span><span class="w"> </span><span class="n">hs</span><span class="w"> </span><span class="n">l</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="n">merkle</span><span class="w"> </span><span class="n">hs</span><span class="w"> </span><span class="n">l</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="n">merkle</span><span class="w"> </span><span class="n">hs</span><span class="w"> </span><span class="n">r</span>
</code></pre></div>
<p>With these two properties, we can define the following type:</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span><span class="w"> </span><span class="nc">imt</span><span class="w"> </span><span class="o">(</span><span class="n">h</span><span class="o">:</span><span class="nn">C</span><span class="p">.</span><span class="n">v_height</span><span class="o">)</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">Type</span><span class="w"> </span><span class="o">=</span>
<span class="w"> </span><span class="n">t</span><span class="o">:</span><span class="n">tree</span><span class="w"> </span><span class="nn">C</span><span class="p">.</span><span class="n">t_C</span><span class="w"> </span><span class="nn">C</span><span class="p">.</span><span class="n">t_H</span><span class="w"> </span><span class="o">{</span><span class="n">incremental</span><span class="w"> </span><span class="n">h</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="n">merkle</span><span class="w"> </span><span class="n">h</span><span class="w"> </span><span class="n">t</span><span class="o">}</span>
</code></pre></div>
<p>So, <code>imt h</code> is the type of incremental Merkle trees of height <code>h</code>. As we’ll see next, this type makes it quite simple and elegant to specify that a tree manipulating function preserves the <span class="caps">IMT</span> invariants directly on the function’s signature.</p>
<p>The third component of the core <span class="caps">IMT</span> model is <code>Tree.Methods</code>, where the functions that operate on IMTs are defined. We’ll focus on insertion, as it’s the only way to modify an <span class="caps">IMT</span>. The <code>insert_list</code> function, whose signature we show next, inserts a list of commitments into a tree. Its implementation is a model of <code>insert</code>, from <code>lib_sapling</code>.</p>
<div class="highlight"><pre><span></span><code><span class="k">val</span><span class="w"> </span><span class="n">insert_list</span><span class="w"> </span><span class="o">:</span>
<span class="w"> </span><span class="n">vs</span><span class="o">:</span><span class="kt">list</span><span class="w"> </span><span class="nn">C</span><span class="p">.</span><span class="n">t_C</span><span class="w"> </span><span class="o">-></span>
<span class="w"> </span><span class="n">pos</span><span class="o">:</span><span class="kt">uint32</span><span class="w"> </span><span class="o">-></span>
<span class="w"> </span><span class="n">h</span><span class="o">:</span><span class="nn">C</span><span class="p">.</span><span class="n">v_height</span><span class="w"> </span><span class="o">-></span>
<span class="w"> </span><span class="n">t</span><span class="o">:(</span><span class="n">imt</span><span class="w"> </span><span class="n">h</span><span class="o">)</span><span class="w"> </span><span class="o">-></span>
<span class="w"> </span><span class="n">Pure</span><span class="w"> </span><span class="o">(</span><span class="n">imt</span><span class="w"> </span><span class="n">h</span><span class="o">)</span>
<span class="w"> </span><span class="o">(</span><span class="n">requires</span><span class="w"> </span><span class="o">(</span><span class="n">length</span><span class="w"> </span><span class="n">vs</span><span class="w"> </span><span class="o"><=</span><span class="w"> </span><span class="n">pow2</span><span class="w"> </span><span class="o">(</span><span class="n">v63</span><span class="w"> </span><span class="n">h</span><span class="o">)</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nn">UInt32</span><span class="p">.</span><span class="n">v</span><span class="w"> </span><span class="n">pos</span>
<span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nn">UInt32</span><span class="p">.</span><span class="n">v</span><span class="w"> </span><span class="n">pos</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">count_leaves</span><span class="w"> </span><span class="n">t</span><span class="o">))</span>
<span class="w"> </span><span class="o">(</span><span class="n">ensures</span><span class="w"> </span><span class="o">(</span><span class="k">fun</span><span class="w"> </span><span class="n">t'</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">to_list</span><span class="w"> </span><span class="n">t'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">to_list</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">@</span><span class="w"> </span><span class="n">vs</span><span class="o">))</span>
<span class="w"> </span><span class="o">(</span><span class="n">decreases</span><span class="w"> </span><span class="o">%[</span><span class="n">v63</span><span class="w"> </span><span class="n">h</span><span class="o">;</span><span class="mi">0</span><span class="o">])</span>
</code></pre></div>
<p>Let’s break down this signature. The first three arguments are quite simple: a list of commitments <code>vs</code>, the position <code>pos</code> of the tree in which those commitments will be inserted, and the height <code>h</code> of the tree (which is bounded by 32). Then, the function takes the initial tree <code>t</code>, which must be an <span class="caps">IMT</span> of height <code>h</code>. The return type first states that the function is <code>Pure</code>, meaning that it terminates and has no side-effects. After that, it describes that the function returns an <span class="caps">IMT</span> of height <code>h</code>. Finally, we have the <code>requires</code> and <code>ensures</code> clauses, which define additional pre- and post-conditions. For pre-conditions, the function requires that the list of commitments fits on the tree, and that <code>pos</code> is the next position to fill. As for post-conditions, we have that the list of elements of the resulting tree must be equal to appending <code>vs</code> to that of <code>t</code>. The <code>decreases</code> tag is just aiding the termination-checker.</p>
<p>It’s important to note that some pre- and post-conditions are implicit (those in the types of the arguments and return value), while others are explicit (<code>requires</code>/<code>ensures</code> clauses). In this example, the implicit properties are used to state the preservation of the <span class="caps">IMT</span> invariants, while the explicit clauses are reserved for the functional specification of <code>insert_list</code>.</p>
<h5 id="insertion-identities"><a class="toclink" href="#insertion-identities">Insertion identities</a></h5>
<p>There were three insertion functions defined in the initial <span class="caps">IMT</span> specification:</p>
<ul>
<li><code>insert_model</code>: a naive implementation that checks if the left sub-tree is full to determine the path,</li>
<li><code>insert_pow</code>: an optimization carrying the next available position,</li>
<li><code>insert_list</code>: a further optimization for batch insertion. This is the one found in the OCaml implementation.</li>
</ul>
<p>In the <code>Tree.InsertionIdentities</code> module the first two are defined, and all three are proven to be semantically equivalent.</p>
<h4 id="full-merkle-tree-translation"><a class="toclink" href="#full-merkle-tree-translation">Full Merkle Tree Translation</a></h4>
<p>As explained in a previous <a href="#incremental-merkle-trees">section</a>, our model uses a compressed representation of IMTs, following their implementation. An important verification target was to prove a translation from this representation to a standard Merkle tree model. To do so, we used the off-the-shelf <a href="https://github.com/FStarLang/FStar/blob/master/examples/data_structures/MerkleTree.fst">Merkle tree model</a> from F*, being able to adapt the Merkle properties proven in this model for our <span class="caps">IMT</span> type.</p>
<h3 id="certified-ocaml-implementation-extraction"><a class="toclink" href="#certified-ocaml-implementation-extraction">Certified OCaml implementation extraction</a></h3>
<p>For the certified OCaml implementation, we need to extract the <code>Tree.Data</code> and <code>Tree.Methods</code> modules. The module <code>ml/Tree.ml</code> bundles these two extracted modules, and should replace the original OCaml implementation.</p>
<p>To perform the extraction, we just ask F* to extract these modules to OCaml. This will generate the OCaml files in <code>_out/</code> containing the extracted code (alongside the <a href="#specification-extraction-and-qcheck-integration">extracted specification</a>).</p>
<p>We can now take a brief look at some of the extracted OCaml code. First, the <code>tree</code> is extracted to the expected OCaml part.</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span><span class="o">,</span> <span class="k">'</span><span class="n">b</span><span class="o">)</span> <span class="n">tree</span> <span class="o">=</span>
<span class="o">|</span> <span class="nc">Empty</span>
<span class="o">|</span> <span class="nc">Leaf</span> <span class="k">of</span> <span class="k">'</span><span class="n">a</span>
<span class="o">|</span> <span class="nc">Node</span> <span class="k">of</span> <span class="k">'</span><span class="n">b</span> <span class="o">*</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span><span class="o">,</span> <span class="k">'</span><span class="n">b</span><span class="o">)</span> <span class="n">tree</span> <span class="o">*</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span><span class="o">,</span> <span class="k">'</span><span class="n">b</span><span class="o">)</span> <span class="n">tree</span>
</code></pre></div>
<p>More interestingly, the refined <code>imt h</code> type becomes:</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="k">'</span><span class="n">h</span> <span class="n">imt</span> <span class="o">=</span> <span class="o">(</span><span class="nn">Commitments</span><span class="p">.</span><span class="n">t_C</span><span class="o">,</span> <span class="nn">Commitments</span><span class="p">.</span><span class="n">t_H</span><span class="o">)</span> <span class="nn">Tree_Data</span><span class="p">.</span><span class="n">tree</span>
</code></pre></div>
<p>Although the refinements are erased as expected, we’re left with a phantom type parameter <code>h</code>, which would be instantiated to <code>unit</code> in the rest of the extracted code. However, as the <code>imt</code> type is tagged with <code>inline_for_extraction</code>, it’s replaced by its definition. As for the <code>insert_list</code> function, the resulting OCaml code is:</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="k">rec</span> <span class="o">(</span><span class="n">insert_list</span> <span class="o">:</span>
<span class="nn">Commitments</span><span class="p">.</span><span class="n">t_C</span> <span class="kt">list</span> <span class="o">-></span>
<span class="nn">Tree_Data</span><span class="p">.</span><span class="n">uint32</span> <span class="o">-></span>
<span class="nn">Commitments</span><span class="p">.</span><span class="n">v_height</span> <span class="o">-></span>
<span class="o">(</span><span class="nn">Commitments</span><span class="p">.</span><span class="n">t_C</span><span class="o">,</span> <span class="nn">Commitments</span><span class="p">.</span><span class="n">t_H</span><span class="o">)</span> <span class="nn">Tree_Data</span><span class="p">.</span><span class="n">tree</span> <span class="o">-></span>
<span class="o">(</span><span class="nn">Commitments</span><span class="p">.</span><span class="n">t_C</span><span class="o">,</span> <span class="nn">Commitments</span><span class="p">.</span><span class="n">t_H</span><span class="o">)</span> <span class="nn">Tree_Data</span><span class="p">.</span><span class="n">tree</span><span class="o">)</span>
<span class="o">=</span>
<span class="k">fun</span> <span class="n">vs</span> <span class="o">-></span>
<span class="k">fun</span> <span class="n">pos</span> <span class="o">-></span>
<span class="k">fun</span> <span class="n">h</span> <span class="o">-></span>
<span class="k">fun</span> <span class="n">t</span> <span class="o">-></span>
<span class="k">match</span> <span class="o">(</span><span class="n">t</span><span class="o">,</span> <span class="o">(</span><span class="nn">Int63</span><span class="p">.</span><span class="n">eq</span> <span class="n">h</span> <span class="o">(</span><span class="mi">0</span><span class="o">)),</span> <span class="n">vs</span><span class="o">)</span> <span class="k">with</span>
<span class="o">|</span> <span class="o">(</span><span class="nn">Tree_Data</span><span class="p">.</span><span class="nc">Empty</span><span class="o">,</span> <span class="bp">true</span><span class="o">,</span> <span class="n">v</span><span class="o">::</span><span class="bp">[]</span><span class="o">)</span> <span class="o">-></span> <span class="nn">Tree_Data</span><span class="p">.</span><span class="nc">Leaf</span> <span class="n">v</span>
<span class="o">|</span> <span class="o">(</span><span class="n">uu___</span><span class="o">,</span> <span class="n">uu___1</span><span class="o">,</span> <span class="bp">[]</span><span class="o">)</span> <span class="o">-></span> <span class="n">t</span>
<span class="o">|</span> <span class="o">(</span><span class="nn">Tree_Data</span><span class="p">.</span><span class="nc">Empty</span><span class="o">,</span> <span class="n">uu___</span><span class="o">,</span> <span class="n">uu___1</span><span class="o">)</span> <span class="o">-></span>
<span class="n">insert_node</span> <span class="o">(</span><span class="nn">Tree_Data</span><span class="p">.</span><span class="n">pred63</span> <span class="n">h</span><span class="o">)</span> <span class="nn">Tree_Data</span><span class="p">.</span><span class="nc">Empty</span>
<span class="nn">Tree_Data</span><span class="p">.</span><span class="nc">Empty</span> <span class="n">pos</span> <span class="n">vs</span>
<span class="o">|</span> <span class="o">(</span><span class="nn">Tree_Data</span><span class="p">.</span><span class="nc">Node</span> <span class="o">(</span><span class="n">uu___</span><span class="o">,</span> <span class="n">l</span><span class="o">,</span> <span class="n">r</span><span class="o">),</span> <span class="n">uu___1</span><span class="o">,</span> <span class="n">uu___2</span><span class="o">)</span> <span class="o">-></span>
<span class="n">insert_node</span> <span class="o">(</span><span class="nn">Tree_Data</span><span class="p">.</span><span class="n">pred63</span> <span class="n">h</span><span class="o">)</span> <span class="n">l</span> <span class="n">r</span> <span class="n">pos</span> <span class="n">vs</span>
</code></pre></div>
<p>Here we see that the final code looks quite similar to the original version, apart from some very clearly automatically generated names for the unused variables in patterns.</p>
<p>The extracted code has important shortcomings, some of which we managed to solve, as explained in a future <a href="#conclusions-and-limitations-of-f--">section</a>.</p>
<p>The <code>Makefile</code> target to run the extraction is:</p>
<div class="highlight"><pre><span></span><code>make extract
</code></pre></div>
<h3 id="specification-extraction-and-qcheck-integration"><a class="toclink" href="#specification-extraction-and-qcheck-integration">Specification extraction and QCheck integration</a></h3>
<p>In our concrete case, we put all the splices in a new module called <code>Tree.PrePost.fst</code>. This is not only helpful for organizing the extracted code, but also allows the user to verify the <span class="caps">IMT</span> model without running the specification extraction tactic (which can be slow).</p>
<p>In turn, running the splices will generate the OCaml file <code>_out/Tree_PrePost.ml</code>, containing the pre- and post-conditions and tests spliced with the <code>extract_spec</code> tactic.</p>
<p>We can take a look at the extracted property for <code>insert_list</code><span class="quo">‘</span>s pre-condition:</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="o">(</span><span class="n">insert_list_pre</span> <span class="o">:</span>
<span class="nn">Commitments</span><span class="p">.</span><span class="n">t_C</span> <span class="kt">list</span> <span class="o">-></span>
<span class="nn">Stdint</span><span class="p">.</span><span class="nn">Uint32</span><span class="p">.</span><span class="n">t</span> <span class="o">-></span>
<span class="nn">Int63</span><span class="p">.</span><span class="n">t</span> <span class="o">-></span>
<span class="o">(</span><span class="nn">Commitments</span><span class="p">.</span><span class="n">t_C</span><span class="o">,</span> <span class="nn">Commitments</span><span class="p">.</span><span class="n">t_H</span><span class="o">)</span> <span class="nn">Tree_Data</span><span class="p">.</span><span class="n">tree</span> <span class="o">-></span> <span class="kt">bool</span><span class="o">)</span>
<span class="o">=</span>
<span class="k">fun</span> <span class="n">vs</span> <span class="o">-></span>
<span class="k">fun</span> <span class="n">pos</span> <span class="o">-></span>
<span class="k">fun</span> <span class="n">h</span> <span class="o">-></span>
<span class="k">fun</span> <span class="n">t</span> <span class="o">-></span>
<span class="o">(((</span><span class="nn">Int63</span><span class="p">.</span><span class="n">gte</span> <span class="n">h</span> <span class="o">(</span><span class="mi">0</span><span class="o">))</span> <span class="o">&&</span>
<span class="o">(</span><span class="nn">Int63</span><span class="p">.</span><span class="n">lte</span> <span class="n">h</span> <span class="nn">Commitments</span><span class="p">.</span><span class="n">max_height</span><span class="o">))</span>
<span class="o">&&</span>
<span class="o">((</span><span class="nn">Tree_Properties</span><span class="p">.</span><span class="n">incremental</span> <span class="n">h</span> <span class="n">t</span><span class="o">)</span> <span class="o">&&</span>
<span class="o">(</span><span class="nn">Tree_Properties</span><span class="p">.</span><span class="n">merkle</span> <span class="n">h</span> <span class="n">t</span><span class="o">)))</span>
<span class="o">&&</span>
<span class="o">(((</span><span class="nn">Util</span><span class="p">.</span><span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">of_int</span> <span class="o"><<</span> <span class="nn">List</span><span class="p">.</span><span class="n">length</span><span class="o">)</span> <span class="n">vs</span><span class="o">)</span> <span class="o"><=</span>
<span class="o">((</span><span class="nn">Prims</span><span class="p">.</span><span class="n">pow2</span> <span class="o">(</span><span class="nn">Tree_Data</span><span class="p">.</span><span class="n">v63</span> <span class="n">h</span><span class="o">))</span> <span class="o">-</span> <span class="o">(</span><span class="nn">Util</span><span class="p">.</span><span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">of_int</span> <span class="o"><<</span> <span class="nn">Stdint</span><span class="p">.</span><span class="nn">Uint32</span><span class="p">.</span><span class="n">to_int</span><span class="o">)</span> <span class="n">pos</span><span class="o">)))</span>
<span class="o">&&</span> <span class="o">((</span><span class="nn">Util</span><span class="p">.</span><span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">of_int</span> <span class="o"><<</span> <span class="nn">Stdint</span><span class="p">.</span><span class="nn">Uint32</span><span class="p">.</span><span class="n">to_int</span><span class="o">)</span> <span class="n">pos</span><span class="o">)</span> <span class="o">=</span> <span class="o">(</span><span class="nn">Tree_Properties</span><span class="p">.</span><span class="n">count_leaves</span> <span class="n">t</span><span class="o">)))</span>
</code></pre></div>
<p>Although it may be somewhat hard to see at once, this is just the conjunction of the refinements in the function’s signature. Note that the <code><<</code> operator is just function composition.</p>
<p>Finally, we want to actually test the certified implementation against the extracted properties. To do so, we first complete the definition of the extracted QCheck generator templates, which can be found in the <code>ml/generators.ml</code>. Next, we copy the extracted QCheck tests in <code>ml/test_internal_tree.ml</code>. For instance, one of these tests is:</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="o">(</span><span class="n">test_insert_model_spec</span> <span class="o">:</span> <span class="nn">FStar_QCheck</span><span class="p">.</span><span class="n">test_t</span><span class="o">)</span> <span class="o">=</span>
<span class="nn">FStar_QCheck</span><span class="p">.</span><span class="n">test_make</span> <span class="s2">"insert_model_spec"</span> <span class="n">gen_insert_model_args</span>
<span class="o">(</span><span class="k">fun</span> <span class="n">a0</span> <span class="o">-></span>
<span class="k">match</span> <span class="n">a0</span> <span class="k">with</span>
<span class="o">|</span> <span class="nn">Prims</span><span class="p">.</span><span class="nc">Mkdtuple2</span> <span class="o">(</span><span class="n">v</span><span class="o">,</span> <span class="nn">Prims</span><span class="p">.</span><span class="nc">Mkdtuple2</span> <span class="o">(</span><span class="n">h</span><span class="o">,</span> <span class="n">t</span><span class="o">))</span> <span class="o">-></span>
<span class="o">(</span><span class="nn">FStar_QCheck</span><span class="p">.</span><span class="n">assume_</span> <span class="o">(</span><span class="n">insert_model_pre</span> <span class="n">v</span> <span class="n">h</span> <span class="n">t</span><span class="o">);</span>
<span class="n">insert_model_post</span> <span class="n">v</span> <span class="n">h</span> <span class="n">t</span>
<span class="o">(</span><span class="nn">Tree_InsertionIdentities</span><span class="p">.</span><span class="n">insert_model</span> <span class="n">v</span> <span class="n">h</span> <span class="n">t</span><span class="o">)))</span>
</code></pre></div>
<p>These splices are executed by running:</p>
<div class="highlight"><pre><span></span><code>make extract
</code></pre></div>
<p>To compile the tests, just run:</p>
<div class="highlight"><pre><span></span><code>make test
</code></pre></div>
<p>This will generate an executable called <code>test.exe</code> in the <code>_out</code> directory, which runs the tests.</p>
<h3 id="testing-the-protocol-sapling-implementation"><a class="toclink" href="#testing-the-protocol-sapling-implementation">Testing the Protocol Sapling Implementation</a></h3>
<p>As mentioned earlier, there are two Sapling implementations in Tezos. Up to now, we’ve solely focused on the one found in <code>lib_sapling</code>, for its relative simplicity. The other implementation is part of the Tezos Protocol, and is written in a stateful manner to make use of the Context (Tezos’ key-value store). This makes this <span class="caps">IMT</span> implementation much harder to verify directly than the <span class="caps">ADT</span>-based version.</p>
<p>This is a perfect opportunity to recall the <em>verification spectrum</em> we proposed in the introduction. The <span class="caps">IMT</span> implementation from <code>lib_sapling</code> sat at one end of this spectrum: it was easy to verify it directly and it could be replaced by the extracted OCaml code. The PBTs extracted from the specification in this case help close a somewhat narrow verification gap (mostly assumptions about machine integers, lists, cryptographic primitive and F*’s extraction mechanism).</p>
<p>The protocol version, on the other hand, stands at the opposite end of the spectrum. Due to its pervasive use of the Context and Lwt (an OCaml library for promises and concurrent I/O with a monadic interface), doing a full verification and replacing the code with the extracted specification becomes unfeasible. Then, the verification effort in this case is limited to a model of the implementation, whose extracted code then becomes a reference implementation against which the complex version can be tested. In our particular case, the work of specifying and verifying an abstract model for IMTs is already done, as we can use the work done for <code>lib_sapling</code> to this end.</p>
<p>Following this idea, we decided to test the protocol Sapling implementation against the certified and tested extracted <span class="caps">ADT</span> implementation. This is implemented on the <code>ml/test/proto-test.ml</code> module.
Several tests were considered, which are discussed in <a href="https://gitlab.com/nomadic-labs/sapling-verification/-/issues/23">this issue</a>.</p>
<p>The key to carrying this out was to define a projection function (<code>project_tree</code>) from the context tree representation to the <span class="caps">IMT</span> <span class="caps">ADT</span>. With this, we can describe the test we implemented with the following diagram:</p>
<p><img src="images/fstar-sapling-pbt/fstar-sapling-cd.png" style="display:block;margin:auto;width:50%;" alt="Testing the Protocol sapling implementation."></p>
<p>This diagram is divided in two by the vertical dotted line that delimits the protocol IMTs to the <span class="caps">ADT</span> IMTs. For instance, t0 stands for a context in which an <span class="caps">IMT</span> is stored, following the representation defined in <code>sapling_storage.ml</code>.</p>
<p>Here, <code>proj0</code> and <code>proj1</code> are calls to the projection function, and <code>insert'</code> is the <span class="caps">IMT</span> insertion function from our <span class="caps">ADT</span> spec (<code>insert_list</code>).</p>
<p>The function that we want to test is <code>insert</code>, the protocol <span class="caps">IMT</span> insertion (actually called <code>add</code>). Our test consists of checking the commutativity of this diagram. This means, to check that, for any given <code>t0</code>, <code>proj1 (insert t0)</code> is equal to <code>insert' (proj0 t0)</code>.</p>
<p>By performing this test we ensure that the complex protocol insertion function is equivalent to the one we know is well-behaved, modulo the projections.</p>
<h2 id="our-experience-of-working-with-f"><a class="toclink" href="#our-experience-of-working-with-f">Our experience of working with F*</a></h2>
<p>In this section we’ll summarize our experience with the F* language, and outline our contributions to the F* project.</p>
<h3 id="general-evaluation-of-f-for-verifying-ocaml-code"><a class="toclink" href="#general-evaluation-of-f-for-verifying-ocaml-code">General evaluation of F* for verifying OCaml code</a></h3>
<p>One aspect that makes F* suitable for verifying OCaml code is the similarity of their syntax and type system. However, F* is not a superset of OCaml, so the modelling of a piece of OCaml code requires some varying degree of manual work. This will depend on the source program, as the translation of certain OCaml constructs (such as functors) is not trivial. In our experience, this process was quite straightforward, but that might not be the case for code bases that make heavy use of first-class modules.</p>
<p>Another of F*’s key selling points is its hybrid nature. The language’s <span class="caps">SMT</span> support did in fact allow for many of our model’s properties to be proven with little to no work, while being flexible enough to spell out more complex proofs. This, however, came at a cost. Overreliance on the <span class="caps">SMT</span> backend often led to increasingly flaky (i.e. unstable) proofs. A small change in a seemingly unrelated lemma might cause a previously proven theorem to no longer be accepted. This issue is often exacerbated by uninformative error messages that require some deciphering, which might negate some of the time savings gained by proof automation. All this should not be read as taking merit away from F*’s hybrid approach, but rather as a remark on the importance of finding the right balance between explicitness and reliance on automation when writing a proof.</p>
<h3 id="contributions-to-f"><a class="toclink" href="#contributions-to-f">Contributions to F*</a></h3>
<p>As previously mentioned, during the duration of this project we encountered some of F*’s rough edges, which are to be expected in a research language. After discussing them with some of the language’s designers we agreed to try to provide solutions for some of these shortcomings, under their guidance. We can split these contributions into two groups.</p>
<h4 id="extraction-of-machine-integers"><a class="toclink" href="#extraction-of-machine-integers">Extraction of machine integers</a></h4>
<p>After going to the trouble of refining the specification with machine <a href="#machine-integers">integer types</a>, it came as a surprise that the extracted OCaml code using machine integers was quite inefficient.</p>
<p>This was mainly due to how machine integer constants were extracted. The code generated for a literal such as <code>42l</code> was the following:</p>
<div class="highlight"><pre><span></span><code><span class="nn">FStar</span><span class="p">.</span><span class="nn">Int32</span><span class="p">.</span><span class="n">int_to_t</span> <span class="o">(</span><span class="nn">Prims</span><span class="p">.</span><span class="n">of_int</span> <span class="mi">42</span><span class="o">)</span>
</code></pre></div>
<p>where <code>int_to_t</code> is defined as:</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="n">int_to_t</span> <span class="n">x</span> <span class="o">=</span> <span class="nn">Stdint</span><span class="p">.</span><span class="nn">Int32</span><span class="p">.</span><span class="n">of_string</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">to_string</span> <span class="n">x</span><span class="o">)</span>
</code></pre></div>
<p>This means that for every machine integer constant, the extracted OCaml code would:</p>
<ol>
<li>Convert it to a Zarith int,</li>
<li>Convert that Zarith int into a string, and</li>
<li>Parse that string to get the desired Stdint int.</li>
</ol>
<p>This added a significant unnecessary overhead, considering that the same can be achieved by the code:</p>
<div class="highlight"><pre><span></span><code><span class="nn">Stdint</span><span class="p">.</span><span class="nn">Int32</span><span class="p">.</span><span class="n">of_int</span> <span class="mi">42</span>
</code></pre></div>
<p>This was precisely the solution we implemented in <a href="https://github.com/FStarLang/FStar/pull/2325"><code>PR FStar/#2325</code></a>, which has already been merged.</p>
<p>A particular case of this issue was the extraction of <code>0</code> and <code>1</code> constants, which followed a similar logic. These cases are special, because all the <code>Stdint</code> modules, which implement F*’s machine integers, expose <code>zero</code> and <code>one</code> constants. So the solution for this was much easier: to add these constants to the machine integer interface. This is implemented by <a href="https://github.com/FStarLang/FStar/pull/2306"><code>PR FStar/#2306</code></a>, which is also merged.</p>
<p>Another source of inefficiency is the <code>Int.Cast</code> module, that defines the conversion between machine integer types. The issue here is almost the same as in the extraction of literals. For casting a value of a machine integer type to another, the extracted code converts the value into a string, parses the string into a Zarith int, which in turn gets converted into a new string that finally gets parsed as a value of the desired type. Again, this means that casting has a significant overhead. This can be avoided by providing OCaml realization for this module, instead of extracting it. The realization can use Stdint casts directly, improving its performance. We implemented this on <a href="https://github.com/FStarLang/FStar/pull/2315"><code>PR FStar/#2315</code></a>, which hasn’t been merged yet.</p>
<h4 id="meta-f"><a class="toclink" href="#meta-f">Meta-F*</a></h4>
<p>The biggest contribution we made to F*, and the most crucial to our project, was adding support for mutually recursive let-bindings to the reflected syntax. Reflection allows inspecting, manipulating and creating F* terms from within F*, and was the tool we used for extracting specifications.</p>
<p>However, F*’s reflection <span class="caps">AST</span> didn’t support mutually recursive let-bindings, so our toolchain wouldn’t work for the function we cared the most about (<code>insert_list</code>).</p>
<p>After discussion with the F* team, and several iterations, we arrived to the solution in <a href="https://github.com/FStarLang/FStar/pull/2291"><code>PR FStar/#2291</code></a>, which has already been merged. This was a (though slight) breaking change, so we also had to fix some uses of the reflection interface in <a href="https://github.com/project-everest/hacl-star/pull/471"><code>PR HACL-star/#471</code></a>.</p>
<p>Finally, we encountered an issue when extracting polymorphic functions built through Meta-F*. In short, if a function’s type had more than one universally quantified type variable, then all except the first one would be instantiated as <code>unit</code> in the extracted code. We solved this issue in <a href="https://github.com/FStarLang/FStar/pull/2305"><code>PR FStar/#2305</code></a>, which is yet to be merged, but has already been reviewed. Although the fix is quite small, finding the source of this error took some time.</p>
<h3 id="unsolved-issues-with-f"><a class="toclink" href="#unsolved-issues-with-f">Unsolved issues with F*</a></h3>
<p>Even though we managed to overcome the limitations of F* outlined in section [ref to section], there’s still one problem with the extracted OCaml code: its reliance on <code>fstarlib</code> (the OCaml implementation of F*’s standard library).</p>
<p>The issue lies in the fact that the extracted code uses <code>fstarlib</code> even when that could be avoided. For instance, primitive types (such as <code>bool</code>) are extracted as their F* standard library symbols (<code>Prims.bool</code>), instead of being extracted directly to their OCaml counterpart. Although both alternatives are semantically equivalent — <code>Prims.bool</code> is defined as a synonym to <code>bool</code> — this means that the extracted OCaml code will depend on <code>fstarlib</code>, even if the only part of the standard library that is used is the primitive types. Then, if one wanted to plug a module extracted form F* into an existing codebase, a number of possibly unnecessary dependencies needed by <code>fstarlib</code> might be added (e.g. <code>batteries</code>, <code>yojson</code>), greatly hindering the adoption of such a workflow.</p>
<p>There are possible, though unsatisfactory, workarounds to this issue. One of them is to manually tinker the extracted code to avoid the uses of the F* types, which is essentially what we did with the <code>cleanup</code> script. It goes without saying that doing this somewhat defeats the purpose of having a certified implementation, so one has to be very careful when doing so.</p>
<h2 id="discussion-and-related-work"><a class="toclink" href="#discussion-and-related-work">Discussion and Related Work</a></h2>
<p>Before we go, we discuss projects that address similar problems through different approaches.</p>
<h3 id="extraction-vs-lightweight-validation"><a class="toclink" href="#extraction-vs-lightweight-validation">Extraction vs Lightweight Validation</a></h3>
<p>Another well-established workflow is to directly extract correct implementations automatically from mechanized implementations. Projects like the <a href="https://vst.cs.princeton.edu/">Verified Software Toolchain</a> narrow the verification gap by building a vertical stack of mechanized components, building on the foundations provided by the <a href="https://compcert.org/">Compcert certified C compiler</a>. Closer to F*, the <a href="https://github.com/FStarLang/karamel">KaRaMeL</a> project allows the extraction of certified programs written in Low*, a subset of F*, to <code>C</code>.</p>
<p>Even when this approach has been successfully adopted in large industrial contexts, such as the verification of the <a href="https://sel4.systems/">seL4 micro-kernel</a>, it would be hard to apply to a codebase like Tezos’, which is naturally designed to evolve. Instead, we propose to tackle the verification gap by leveraging the automated extraction of property-based tests from a formal specification in order to validate that the implementation complies with their formal model.</p>
<h3 id="quickchick"><a class="toclink" href="#quickchick">QuickChick</a></h3>
<p><a href="https://softwarefoundations.cis.upenn.edu/qc-current/index.html">QuickChick</a> is a randomized property-based testing plugin for the Coq Proof Assistant. Its central idea is <em>foundational testing</em>, which means that the testing code is formally verified to be testing the desired property. Additionally, QuickChick supports automatic derivation of generators for data satisfying a particular predicate. Our approach is more light-weight: implementing QuickChick’s features for F* would have required a very significant effort, and QCheck was already integrated into the Octez test suite.</p>
<p>There has also been some <a href="https://icfp21.sigplan.org/details/ocaml-2021-papers/12/Property-Based-Testing-for-OCaml-through-Coq">recent work</a> leveraging QuickChick for testing OCaml code. Their workflow is the opposite of ours: they use <a href="https://github.com/foobar-land/coq-of-ocaml"><code>coq-of-ocaml</code></a> to translate OCaml definitions into Coq, and then rely on QuickChick to test the implementation. Moreover, this approach suffers from the limitation that <code>coq-of-ocaml</code> cannot fully translate all OCaml features into Gallina (Coq) without making arbitrary design decisions which carry
semantic weight.</p>
<h3 id="monolith"><a class="toclink" href="#monolith">Monolith</a></h3>
<p><a href="https://gitlab.inria.fr/fpottier/monolith">Monolith</a> is a framework for testing OCaml libraries, that supports random testing and fuzzing. The user has to specify the library’s interface (types and operations) and provide a reference implementation. Then, Monolith runs sequences of operations trying to find unexpected behaviours.</p>
<p>Given that we propose extracting reference implementations from formally verified abstract models, it would be interesting to study if those could be integrated with Monolith. This way, the open issue of defining generators could be solved by making use of Monolith’s fuzzing and random testing features.</p>
<h3 id="extracting-effectful-specifications"><a class="toclink" href="#extracting-effectful-specifications">Extracting effectful specifications</a></h3>
<p>Currently, our tool targets only <em>pure</em> F* programs with Boolean pre- and post-conditions. An interesting line of future work would be to extend it to support other F* monadic effects, and to leverage the specification extraction mechanism to test arbitrary monadic OCaml code implementing such effects. For example, in our setting this could be used for directly extracting a specification for the more complex stateful implementation of Incremental Merkle Trees.</p>Two weeks at the OPLSS 2022 — and some reflections on the elegance of Call-by-Push-Value2022-08-03T12:00:00+02:002022-08-03T12:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-08-03:/oplss22-retrospective.html<p>A short summary of our experience at <span class="caps">OPLSS</span> this year.</p><p><em>Nomadic Labs PhD student Colin Gonzalez attended the Oregon
Programming Language Summer School, that had the Tezos Foundation as a
sponsor in 2022. In this blog post he reflects on his experiences.</em></p>
<p>One of the perks of preparing a PhD is that sometimes you get to
travel to conferences or summer schools. A summer school is like a
summer camp where the main activity is attending lectures with fellow
grad students from around the world. And on your free time you gather
with people to grab a couple of drinks and discuss the lectures but
also your own PhD experience. I recently got to attend this year’s
edition of <a href="https://www.cs.uoregon.edu/research/summerschool/summer22/">Oregon Programming Language Summer
School</a> in
Eugene, <span class="caps">OR</span>.</p>
<p><span class="caps">OPLSS</span> usually lasts two weeks. The first one covers a set of
introductory courses on proof theory, type theory and logic; the
second week, selected lecturers among the top researchers in the
programming languages field are invited to discuss more advanced
topics in a privileged setting where students and lecturers share the
campus in very casual fashion. This creates opportunities for great
discussions over dinner and lunch.</p>
<p>The first few days were rhythmed by a combination of proof theory by
<a href="http://www.cs.cmu.edu/~fp/">Frank Pfenning</a>, type theory by <a href="http://www.cs.nott.ac.uk/~psztxa/">Thorsten
Altenkirch</a>, algebraic programming
by <a href="https://www.cs.ox.ac.uk/people/jeremy.gibbons/">Jeremy Gibbons</a>
and game semantics by <a href="https://www.irif.fr/~curien/">Pierre-Louis
Curien</a>. Eventually the week closed with
a great course on classical realizability by <a href="https://www.pauldownen.com/">Paul
Downen</a> and rewriting theory with <a href="http://imft.ftn.uns.ac.rs/~silvia/Main">Sylvia
Ghilezan</a>.</p>
<p>Admittedly, one of the most difficult lectures was the one on Game
Semantics. A lecture during which you would glance at your neighbour
with a puzzling face and asking quietly: “Do you understand anything?”.
The answer would often be just “No” and the same puzzling face. It
was for most of us a new topic.</p>
<p>All these topics, either new or a refresh, were a great preparation
for the hours we would spend with <a href="http://www.cs.cmu.edu/~rwh/">Robert
Harper</a> discussing Logical Relations or
implementing a dependent-type theory type-checker with <a href="https://www.cis.upenn.edu/~sweirich/">Stephanie
Weirich</a>. Also, it was the
opportunity to have <a href="https://www.cs.cmu.edu/~balzers/">Stephanie
Balzer</a> walking us through the world
of Pi-Calculus and Session Types and hearing <a href="http://adam.chlipala.net/">Adam
Chlipala</a> explain how to use a proof
assistant like Coq to certify production grade software and
hardware. Finally the lectures of <a href="https://homepages.inf.ed.ac.uk/slindley/">Sam
Lindley</a> and <a href="https://www.cis.upenn.edu/~stevez/">Steve
Zdancevic</a> were a great pair
talking respectively of effect handlers and using the free monad in
DeepSpec to formalise and verify imperative programs.</p>
<p>I noticed that the lecturers kept connecting
<cite>Call-by-Push-Value[1]</cite> with linear logic and type-theory.
Call-by-Push-Value, <span class="caps">CBPV</span> for short, which plays a central role in my
PhD research, happens to be a very active topic in the <span class="caps">US</span>, and it was
a pleasant surprise to get so much new insight into this domain during the
conference. <span class="caps">CBPV</span> is a formalisation of the connection between values
and computations. It was an amazing time to get a new perspective on
my work.</p>
<p>We can observe this duality when comparing OCaml and Haskell. While
they share similar semantics and features, they differ in the way they
treat values. We could say that in Haskell, every program is a value
in its own right. Whereas in OCaml, values are programs that are in a
special form that can’t be reduced any further. Take for instance the
very short OCaml program <code>(fun x -> x) (1 + 1)</code>: OCaml will first
replace <code>1 + 1</code> with the value <code>2</code>, before handing it to the function,
while Haskell would happily pass <code>(1 + 1)</code> as an unevaluated argument
to the function. Eventually, they both give the same result. But this
tiny difference has important implications: OCaml and Haskell programs
do not behave the same because computations happen in different
orders. And this is where <span class="caps">CBPV</span> shines: it can be seen as a core
language capable of interpreting both OCaml and Haskell programs correctly.</p>
<p>And the obvious question arises: why is <span class="caps">CBPV</span> of interest while doing a
PhD at Nomadic Labs? In my PhD, I work on compiling spreadsheets, considered as
programs, to Michelson smart contracts, and I use <span class="caps">CBPV</span> as a formal
framework to reason about the semantics of these programs.</p>
<p>Overall, <span class="caps">OPLSS</span> 2022 was a great experience. I got to enjoy some of the best <span class="caps">CS</span>
lectures I have ever attended, and I gained new perspectives and
insight on my own work.</p>
<p>[1]Paul Blain Levy. 1999. Call-by-Push-Value: A Subsuming Paradigm.</p>Announcing Tezos’ 11th protocol upgrade proposal, “Kathmandu”2022-07-13T22:00:00+02:002022-07-13T22:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-07-13:/announcing-tezos-11th-protocol-upgrade-proposal-kathmandu.html<p>A joint post from Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda, Functori <span class="amp">&</span> Tweag.</p><p><em>This is a joint post from Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda, Functori <span class="amp">&</span> Tweag.</em></p>
<p>We were proud to see Jakarta <a href="https://research-development.nomadic-labs.com/jakarta-is-live.html">go live</a> on June 28th, 2022. In keeping with our policy of proposing upgrades on a regular schedule, we’re happy to announce our next Tezos protocol proposal, Kathmandu. As usual, Kathmandu’s true name is its hash, which is <code>PtKathmankSpLLDALzWw7CGD2j2MtyveTwboEYokqUCP4a1LxMg</code>.</p>
<p>The main features are:</p>
<ul>
<li><a href="https://research-development.nomadic-labs.com/next-generation-rollups.html">Smart contract optimistic rollups</a> enabled on bleeding edge testnets</li>
<li><a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html#first-steps-towards-pipelined-block-validation">Pipelined validation</a> of manager operations (ongoing)</li>
<li><a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html#improved-randomness-using-verifiable-delay-functions">Improved randomness</a> with integration of Verifiable Delay Functions</li>
<li><a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html#support-for-permanent-test-networks">Support for tailored governance</a> for permanent testnets (<a href="https://teztnets.xyz/ghostnet-about"><code>Ghostnet</code></a>)</li>
<li><a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html#event-logging-in-tezos-smart-contracts">Event logging</a> in Michelson smart contracts</li>
<li><a href="https://tezos.gitlab.io/protocols/014_kathmandu.html#increase-paid-storage">New operation for increasing paid storage</a> of a smart contract</li>
</ul>
<p>For more details, see our <a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html">Kathmandu preview post</a>. A deeper technical description can be found in the protocol proposal’s <a href="https://tezos.gitlab.io/kathmandu/protocol.html">technical documentation</a>, and a complete list of changes is provided in the <a href="https://tezos.gitlab.io/protocols/014_kathmandu.html">changelog</a>.</p>
<h3 id="continued-efforts-to-strengthen-testing"><a class="toclink" href="#continued-efforts-to-strengthen-testing">Continued efforts to strengthen testing</a></h3>
<p>As always, we encourage the community to participate in <a href="https://teztnets.xyz/">Tezos testnets</a>, which is of great help in the protocol development process. There are different options:</p>
<h4 id="protocol-testing-on-kathmandunet"><a class="toclink" href="#protocol-testing-on-kathmandunet">Protocol testing on Kathmandunet</a></h4>
<p>We are happy to see <a href="https://www.walletbeacon.io/">Beacon</a> and <a href="https://tezostaquito.io/">Taquito</a> having already added support for <a href="https://teztnets.xyz/kathmandunet-about"><code>Kathmandunet</code></a>. It is critical to have as many bakers and builders as possible participating in this testnet, by running nodes, producing blocks and deploying apps and infrastructure.</p>
<p>Nomadic Labs will publish a release candidate for a new Octez suite version, <code>v14.0~rc1</code>, within a few days. It will include the daemon binaries that enable participation in the <a href="https://teztnets.xyz/kathmandunet-about"><code>Kathmandunet</code></a> test network.</p>
<p>Should the Kathmandu protocol proposal be accepted by the community, v14 of Octez (or later) will be required to participate in consensus due to necessary changes introduced to the protocol environment.</p>
<h4 id="smart-contract-rollup-testing-on-bleeding-edge-testnets"><a class="toclink" href="#smart-contract-rollup-testing-on-bleeding-edge-testnets">Smart contract rollup testing on bleeding edge testnets</a></h4>
<p>The <a href="https://research-development.nomadic-labs.com/next-generation-rollups.html">smart contract optimistic rollups</a> core business logic and Wasm <span class="caps">PVM</span> are in a mature enough state for the community to start developing and testing applications and infrastructure on testnets.</p>
<p>As mentioned in our <a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html">Kathmandu preview post</a>, smart contract rollups will not be enabled on Tezos mainnet by the Kathmandu protocol proposal. This is to give the community extra time to develop and test infrastructure and Layer 2 applications.</p>
<p>Given that <code>Kathmandunet</code> is meant to mirror the current protocol proposal, it will not have support for smart contract rollups enabled. Instead, we encourage the community to start building and testing on the bleeding edge <a href="https://teztnets.xyz/mondaynet-about"><code>Mondaynet</code></a> and <a href="https://teztnets.xyz/dailynet-about"><code>Dailynet</code></a> testnets.</p>
<h4 id="continuous-dapp-testing-on-ghostnet"><a class="toclink" href="#continuous-dapp-testing-on-ghostnet">Continuous dApp testing on Ghostnet</a></h4>
<p><a href="https://teztnets.xyz/ghostnet-about"><code>Ghostnet</code></a> is live! It’s a permanent testnet that follows Mainnet upgrades, meaning dApp developers will no longer have to redeploy to a new testnet after each upgrade. It was activated with a User Activated Upgrade (Ithacanet -> Jakarta), and another <span class="caps">UAU</span> will be necessary to migrate to Kathamandu. After that, further evolution of <code>Ghostnet</code> will happen via a <a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html#support-for-permanent-test-networks">new upgrade mechanism</a> managed by Oxhead Alpha.</p>
<p>Further details are available in the <a href="https://gitlab.com/tezos/tzip/-/blob/703c4a23421cdf860b178fbe8b4b7eb88c33572d/drafts/current/draft-testnet-admin-key.md"><span class="caps">TZIP</span></a> advocating for this feature. The implementation is a contribution of <a href="https://gitlab.com/g.b.fefe">G.-B. Fefe</a>, a community member not affiliated with the core developing teams behind this protocol proposal. For this work, an invoice of 3000 <span class="caps">XTZ</span> is included in the proposal.</p>
<p>We are interested in the communty’s input on the optimal date for testing migration of <code>Ghostnet</code> to the next Mainnet protocol. The migration from Ithaca to Jakarta was performed a few hours before Mainnet activation. For Kathmandu, we are considering to do <code>Ghostnet</code> migration 72 hours prior to Mainnet activation, should the proposal be adopted.</p>
<p>However, we would like to empirically figure out the best time (during the Adoption period) to perform the migration. The community’s participation in the test network and feedback will be most welcome in this process.</p>Why the next generation of optimistic rollups are a game-changer for Tezos2022-07-11T14:00:00+02:002022-07-11T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-07-11:/next-generation-rollups.html<p>It’s not just about smart contracts and scaling. The next generation of optimistic rollups are a plug’n’play solution for running any software on Tezos</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: It’s not just about smart contracts and scaling. The next generation of optimistic rollups are a plug’n’play solution for running any software on Tezos.</strong></p>
<p><em>This is a joint post from Nomadic Labs, TriliTech, Tarides, and Functori.</em></p>
<p>With the Jakarta upgrade and Transactions Optimistic Rollups successfully activated, it’s time to look ahead to the next major step in Tezos protocol development: Smart Contract Optimistic Rollups (SCORUs).</p>
<p>These next-generation optimistic rollups don’t just enable smart contracts. They are a platform for running any type of software, including emulating the Ethereum Virtual Machine (<span class="caps">EVM</span>), and having all computation made verifiable on the Tezos blockchain. That is a game changer for Tezos functionality.</p>
<p>As explained in the recent preview of the <a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html">upcoming ‘Kathmandu’ protocol upgrade proposal</a>, they will hit test networks soon, while mainnet activation is expected to be included in ‘L’ protocol upgrade proposal.</p>
<p>For a general understanding of optimistic rollups and the rationale behind them as a scaling solution for Tezos, check out our <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html">blog post on this topic</a>. See also our blog post on <a href="https://research-development.nomadic-labs.com/toru-introduction.html">Transaction Optimistic Rollups</a> (TORUs).</p>
<p>Let’s jump in.</p>
<h2 id="from-transactions-to-general-computation"><a class="toclink" href="#from-transactions-to-general-computation">From transactions to general computation</a></h2>
<p>Where TORUs have a fixed design focused on enabling higher transaction throughput while being highly decentralized, SCORUs can do whatever they are programmed to. Executing smart contracts is just one of many use cases. In fact, <em>general computation optimistic rollups</em> might have been a better description, but here we are.</p>
<p>For this, a different rollup dispute mechanism is needed.</p>
<p>TORUs are limited to transactions due to a simpler design of this mechanism. If a commitment published by a node operator is wrong, an honest operator can neutralize it by broadcasting a single L1 operation containing proof.</p>
<p>This one-step procedure means the proof must be small enough to be included in a single L1 operation, putting a limitation on the number and complexity of operations you can do between two commitments.</p>
<p>Smart contract rollups remove this limitation by using a different, more advanced dispute mechanism. Rather than being settled in one operation, it works as an <em>interactive</em> process between two rollup node operators.</p>
<p>This <em>refutation game</em> plays out over multiple rounds and Layer 1 blocks. The parties start broad and step by step narrow down the area of contention. When the exact point is found, and the required proof is small enough to be included in a Layer 1 block, the dispute can be resolved.</p>
<p>The bottom line: Interactive proofs remove the limit to the complexity of rollup operations that can be handled by Layer 1, opening the door for general computation.</p>
<table cellpadding="15">
<tr>
<td style="border: 1px solid black; text-align: center;">
</td>
<td style="border: 1px solid black; text-align: center;"><strong>Transaction rollup</strong>
</td>
<td style="border: 1px solid black; text-align: center;"><strong>Smart contract rollup</strong>
</td>
</tr>
<tr>
<td style="border: 1px solid black; text-align: center;"><strong>Functionality</strong>
</td>
<td style="border: 1px solid black; text-align: center;">Asset transfers (simple)
</td>
<td style="border: 1px solid black; text-align: center;">General computation (complex)
</td>
</tr>
<tr>
<td style="border: 1px solid black; text-align: center;"><strong>Who can be a node operator</strong>
</td>
<td style="border: 1px solid black; text-align: center;">Anyone
</td>
<td style="border: 1px solid black; text-align: center;">Anyone
</td>
</tr>
<tr>
<td style="border: 1px solid black; text-align: center;"><strong>User restrictions</strong>
</td>
<td style="border: 1px solid black; text-align: center;">None
</td>
<td style="border: 1px solid black; text-align: center;">Configurable at deployment
</td>
</tr>
<tr>
<td style="border: 1px solid black; text-align: center;"><strong>Admin rights</strong>
</td>
<td style="border: 1px solid black; text-align: center;">None
</td>
<td style="border: 1px solid black; text-align: center;">Configurable at deployment
</td>
</tr>
<tr>
<td style="border: 1px solid black; text-align: center;"><strong>Dispute mechanism</strong>
</td>
<td style="border: 1px solid black; text-align: center;">Single step (non-interactive)
</td>
<td style="border: 1px solid black; text-align: center;">Refutation game (interactive)
</td>
</tr>
<tr>
<td style="border: 1px solid black; text-align: center;"><strong>Operation complexity</strong>
</td>
<td style="border: 1px solid black; text-align: center;">Limited
</td>
<td style="border: 1px solid black; text-align: center;">Unbounded
</td>
</tr>
</table>
<h2 id="the-rollup-computer"><a class="toclink" href="#the-rollup-computer">The rollup “computer”</a></h2>
<p>To understand the design of SCORUs, think of a computer connected to a network.</p>
<p>The rollup itself is like an <span class="caps">IP</span> address on the network, through which users can access the computer “hardware” – a virtual machine run by rollup node operators.</p>
<p>The hardware has no inherent functionality. All functionality comes with an operating system that needs to be installed first. This we call a <em>kernel.</em></p>
<p>A kernel can be an Ethereum Virtual Machine (<span class="caps">EVM</span>) emulation, enabling Solidity smart contracts. It can also be a simpler, single-application kernel that focuses on, e.g., transactions of assets. Or something completely different. As such, there are different ways to design a rollup “stack”.</p>
<table cellpadding="15">
<tr>
<td colspan="3" style="border: 1px solid black; text-align: center;"><strong>The Tezos rollup “stack”</strong>
</td>
</tr>
<tr>
<td style="border: 1px solid black; text-align: center;"><strong><em>Computer analogy</em></strong>
</td>
<td style="border: 1px solid black; text-align: center;"><strong><span class="caps">EVM</span> rollup example</strong>
</td>
<td style="border: 1px solid black; text-align: center;"><strong>Transaction rollup example</strong>
</td>
</tr>
<tr>
<td style="border: 1px solid black; text-align: center;">
</td>
<td style="border: 1px solid black; text-align: center;">Solidity smart contracts
</td>
<td style="border: 1px solid black; text-align: center;">
</td>
</tr>
<tr>
<td style="border: 1px solid black; text-align: center;"><em>Operating system </em>
</td>
<td style="border: 1px solid black; text-align: center;">Kernel (<span class="caps">EVM</span> engine)
</td>
<td style="border: 1px solid black; text-align: center;">Kernel (Transaction engine)
</td>
</tr>
<tr>
<td style="border: 1px solid black; text-align: center;"><em>Hardware</em>
</td>
<td style="border: 1px solid black; text-align: center;">Virtual Machine (Wasm)
</td>
<td style="border: 1px solid black; text-align: center;">Virtual Machine (Wasm)
</td>
</tr>
<tr>
<td style="border: 1px solid black; text-align: center;"><em><span class="caps">IP</span> address</em>
</td>
<td style="border: 1px solid black; text-align: center;">Rollup address
</td>
<td style="border: 1px solid black; text-align: center;">Rollup address
</td>
</tr>
<tr>
<td style="border: 1px solid black; text-align: center;"><em>Network</em>
</td>
<td style="border: 1px solid black; text-align: center;">Tezos blockchain
</td>
<td style="border: 1px solid black; text-align: center;">Tezos blockchain
</td>
</tr>
</table>
<p>At the time of launch, we will make a simple kernel with transaction functionality similar to TORUs available for demonstration purposes. A proof-of-concept kernel in the form of a token exchange using <a href="https://www.marigold.dev/post/bls-signatures"><span class="caps">BLS</span> signatures</a> will also be available. The latter is meant to be integrated with the in-development <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html#next-level-scaling">data-availability layer</a> to demonstrate the next-level scaling potential of Tezos. We are also actively working on an <span class="caps">EVM</span> kernel, though with a release date that is yet to be determined, and a Michelson kernel is being researched.</p>
<p>As with an operating system, kernels can be updated or replaced at any time, though it depends on how the host rollup is configured. Staying with the server hardware analogy, these rollups come with a kind of boot sector which defines a number of its properties at the time of deployment, such as who can update the kernel, if anyone.</p>
<h2 id="the-proof-generating-virtual-machine"><a class="toclink" href="#the-proof-generating-virtual-machine">The Proof-generating Virtual Machine</a></h2>
<p>With all this talk of hardware and operating systems, keep in mind that the whole point of rollups is to <em>execute off-chain</em> and <em>verify on-chain</em>.</p>
<p>Rollup operations are processed off-chain, on Layer 2, by dedicated rollup nodes. They continuously post commitments on the Tezos main chain, Layer 1, representing updates to the state of the rollup.</p>
<p>As it works with optimistic rollups, commitments are treated by default as correct but can be disputed within a given timeframe by anyone else running a rollup node. For those situations, a <em>Proof-generating Virtual Machine</em> (<span class="caps">PVM</span>) is implemented in the Tezos protocol. It’s a slightly modified virtual machine that can output a proof that operations have been processed correctly. The rollup node can use this implementation to produce a proof and post it to Layer 1, where it will be checked by the Layer 1 nodes.</p>
<p>This <em>on-chain</em> process is only activated when a dispute needs to be resolved, and only for a small execution step, once the exact point of contention has been pinpointed through a refutation game.</p>
<h2 id="webassembly-as-the-initial-base-layer"><a class="toclink" href="#webassembly-as-the-initial-base-layer">WebAssembly as the (initial) base layer</a></h2>
<p>The first virtual machine, and <span class="caps">PVM</span>, for Tezos rollups will run <a href="https://webassembly.org/">WebAssembly</a>, or Wasm – a low-level assembly-like language, which is well on track to become a standard for high-performance applications on the web. It’s designed as a compilation target for other languages, much like how desktop programs are compiled into a binary format.</p>
<p>Starting with a Wasm <span class="caps">VM</span> means Tezos will instantly be able to welcome a much larger developer community by supporting a number of popular programming languages. Notably C, C++, and Rust all have good Wasm-compilers. Support for blockchain specific languages like Tezos’ Michelson and Ethereum’s Solidity can be added through kernels or with new (proof-generating) virtual machines added in future protocol upgrades.</p>
<p>For integrating Wasm, we had the advantage that the <em><a href="https://github.com/WebAssembly/spec/tree/master/interpreter">reference interpreter</a></em> of Wasm is written in the same language used for the Tezos protocol: OCaml! For any programming language, you need to specify how it works. One way to do this is to program a virtual machine declaratively – that is, in a way that makes it easy for others to understand how it’s meant to work by simply reading the code. That is a reference interpreter.</p>
<p>So we have been able to integrate <em>the</em> <em>official definition of Wasm</em> into Tezos – with some modifications enabling it to produce proofs about its execution.</p>
<h2 id="calling-all-builders"><a class="toclink" href="#calling-all-builders">Calling all builders</a></h2>
<p>Some kernels will be provided by protocol developer teams, but we encourage ecosystem builders to start developing their own, innovative kernels.</p>
<p>For this we provide a native token bridge based on <a href="https://www.marigold.dev/post/tickets-for-dummies">Michelson tickets</a>, similar to the one used for TORUs, and a bare-bones execution environment supporting token exchanges between Layer 1 and Layer 2, interaction with Layer 1 smart contracts, gas monitoring, and self-governance.</p>
<p>We also provide safe Rust bindings to the Wasm <span class="caps">PVM</span>. The point is to let kernel developers think less about I/O on the blockchain, how blocks work etc., and instead provide an abstract layer that makes it more like working with files as in a standard computer program.</p>
<p>The kernel has access to the full state of the rollup (Layer 2), but any I/O with Layer 1 has to go through the rollups inbox/outbox, asynchronously. For this task, the bindings are like an <span class="caps">API</span>, so, for instance, if you want to send an asset from the rollup to a Layer 1 address, there’s a ready-made function for that.</p>
<p>With these bindings, developers can write a kernel, compile it to WebAssembly, and know that its interaction with the Tezos blockchain will be reliable and secure. We chose Rust because it is a popular language with a mature toolchain providing robust compilation to Wasm.</p>
<h2 id="opening-up-tezos"><a class="toclink" href="#opening-up-tezos">Opening up Tezos</a></h2>
<p>With this design of optimistic rollups, Tezos opens itself up to interacting with all kinds of software systems and making their operation verifiable on the Tezos blockchain. This puts Tezos solidly at the forefront of blockchain technology.</p>
<p>Wasm establishes a highly flexible and future-proof base layer that doesn’t exclude support for currently popular execution environments like the <span class="caps">EVM</span>, or the well known security-focused environment of Michelson, while still inviting Rust and C/C++ developers on board.</p>
<p>Each rollup will be highly configurable, and will have governance modules for kernel upgrades. In this sense, a rollup will offer everything that parachains, subnets, app-chains and similar scaling solutions used by other blockchains do.</p>
<p>The difference from most of these is that rollups on Tezos offer all that while being secured by a time-tested, highly decentralized and censorship resistant Layer 1: the Tezos main chain.</p>
<p>We believe this to be the future of blockchains, and we are excited to be building it with the Tezos community!</p>Jakarta, the latest Tezos upgrade, is LIVE!2022-06-29T01:00:00+02:002022-06-29T01:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-06-29:/jakarta-is-live.html<p>Tezos has activated its tenth upgrade to the protocol, Jakarta.</p><p>On 28 June 2022 23:31 <span class="caps">CET</span>, the Tezos blockchain successfully upgraded by activating the <a href="https://research-development.nomadic-labs.com/announcing-tezos-10th-protocol-upgrade-proposal-jakarta.html">Jakarta proposal</a> (technically <a href="https://research-development.nomadic-labs.com/announcing-jakarta-two.html">Jakarta 2</a>) at block <a href="https://tzkt.io/2490369">#2490369</a>.</p>
<p>This tenth Tezos protocol upgrade was jointly developed by Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda, Functori <span class="amp">&</span> Tweag.</p>
<p>We were happy to observe 2/3 of the total stake having performed the required upgrade to Octez v13 or TezEdge v3 a day before activation. Still, we encourage bakers to upgrade to new required node versions at the earliest possible time. This minimizes the risk of the network getting stuck (or advancing slowly) due to the consensus requirements introduced with <a href="https://research-development.nomadic-labs.com/ithaca-is-live.html">Tenderbake</a>.</p>
<p>Further updates to the protocol environment, and hence mandatory node and baker upgrades, <em>will</em> be necessary for features in upcoming upgrade proposals, such as Smart Contract Optimistic Rollups. We greatly appreciate the community’s help and efforts in keeping Tezos at the forefront of blockchain innovation.</p>
<p>Now, let’s celebrate the new features and improvements included in the Jakarta protocol! Not least the first scaling solution on Tezos, and one of few truly decentralized Layer 2 solutions out there: <strong>Transaction Optimistic Rollups (TORUs)</strong>.</p>
<p>TORUs play the important role of:</p>
<ul>
<li>addressing short-term scaling needs;</li>
<li>demonstrating Tezos’ ability to implement scaling solutions through protocol upgrades;</li>
<li>enabling ecosystem developers to build rollup infrastructure;</li>
<li>paving the way for Smart Contract Optimistic Rollups, coming in a future proposal.</li>
</ul>
<p>For more on rollups, see our blog posts <a href="https://research-development.nomadic-labs.com/toru-introduction.html">introducing TORUs</a> and <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html">outlining the rollup-based scaling strategy</a>. Those interested in running a rollup node can check out <a href="https://www.marigold.dev/post/running-rollup-node-to-test-toru-on-jakartanet">this rollup tutorial</a>.</p>
<p>Other changes in Jakarta include:</p>
<ul>
<li>
<p><strong>Tickets hardening:</strong> The protocol now explicitly tracks ownership of <a href="https://tezos.gitlab.io/jakarta/michelson.html#operations-on-tickets">Michelson tickets</a> (see also ‘<a href="https://www.marigold.dev/post/tickets-for-dummies">Tickets for Dummies</a>’) by checking ticket creation and ownership changes against a global balance table. This adds extra protection against attempts to forge tickets and increases security for Layer 2 solutions that use tickets to represent Layer 1 assets (e.g., TORUs). With this change, tickets are no longer <a href="https://research-development.nomadic-labs.com/edo-the-latest-tezos-upgrade-is-live.html">considered experimental</a> and are believed safe for use on mainnet.</p>
</li>
<li>
<p><strong>A safer Sapling integration:</strong> The integration of Sapling transactions into Michelson smart contracts has been changed to <a href="https://research-development.nomadic-labs.com/fixing-the-sapling-protocol-integration.html">a new, safer design</a>. With Jakarta’s activation it is now only possible to originate Sapling smart contracts which conform to the new version.</p>
</li>
<li>
<p><strong>New Liquidity Baking voting:</strong> The Liquidity Baking Escape Hatch is now a “Liquidity Baking Toggle Vote”, with options <em>On, Off,</em> or <em>Pass</em>. Furthermore, a deactivation is no longer permanent and can be reversed by a later change in votes. More information <a href="https://gitlab.com/tezos/tzip/-/blob/master/drafts/current/draft-symmetric-liquidity-baking-toggle-vote.md">here</a>.</p>
</li>
<li>
<p><strong>Michelson interpreter improvements:</strong> In particular, type safety and performance <a href="https://research-development.nomadic-labs.com/also-coming-with-jakarta-spring-cleaning-the-michelson-interpreter.html">has been improved</a>. A few smart contracts which relied on legacy features have been patched to be compliant with the modern Michelson specification.</p>
</li>
<li>
<p><strong>Rolls are no more:</strong> The Jakarta protocol proposal redefines the computation of delegates’ voting power in the self-amendment process. Instead of being measured in terms of rolls, it is now directly proportional to a delegate’s stake.</p>
</li>
</ul>
<p>As always, the upgrade train doesn’t stop here. We are working hard on finalizing the upcoming <a href="https://research-development.nomadic-labs.com/from-jakarta-to-kathmandu-non-stop.html">Kathmandu proposal</a> – and on further laying the groundwork for Smart Contract Optimistic Rollups, a Data Availability Layer, and other great improvements to Tezos.</p>From Jakarta to Kathmandu non-stop2022-06-23T15:00:00+02:002022-06-23T15:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-06-23:/from-jakarta-to-kathmandu-non-stop.html<p>Jakarta protocol is activating soon — here’s what’s coming next.</p><!---
Draft Summaries:
https://hackmd.io/b97-frwETCG429b5G3S05Q
Online Preview:
https://hackmd.io/2TQT_N2DTriuzRHGKTH2TQ
---->
<p><em>This is a joint post from Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda, Functori <span class="amp">&</span> Tweag.</em></p>
<p>The Tezos community is preparing for exciting new developments in the <a href="https://tzstats.com/election/head">coming days</a>: the activation of the <a href="https://research-development.nomadic-labs.com/announcing-jakarta-two.html">Jakarta</a> protocol upgrade will be the first step <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html">towards massive scalability</a>, with the introduction of the first Tezos Layer 2 solution: <a href="https://research-development.nomadic-labs.com/toru-introduction.html">Transaction Optimistic Rollups</a>, or TORUs.</p>
<p>Today we look a bit further ahead. However beautiful and lively a port Jakarta is, the Tezos protocol’s path to scalability doesn’t stop there. After mending our sails and reloading our batteries, we are eager to present to the community what’s next in our pipeline. We hope you brought your mountain gear with you – next stop is Kathmandu.</p>
<p>In this article we give you a preview of the different projects that have kept us busy in the last months. We will describe the <a href="https://gitlab.com/tezos/tezos/-/milestones/79#tab-issues">features</a> which, after some stabilization in the coming weeks, are likely to be part of Kathmandu, Tezos’ 11th protocol upgrade proposal. But, first we will focus on Smart Contract Optimistic RollUps (aka SCORUs), our next Layer 2 scalability solution, and how big new features like this one require us to rethink the development process for protocol upgrades.</p>
<h2 id="scorus-are-rolling-into-testsnets"><a class="toclink" href="#scorus-are-rolling-into-testsnets">SCORUs are rolling into testsnets</a></h2>
<p>As we hinted in an earlier <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html">scalability preview</a>, SCORUs provide a generic infrastructure to implement any computational device, as long as its semantics can be described as a Proof-producing Virtual Machine (<span class="caps">PVM</span>), in the Tezos protocol.</p>
<p>This design allows us to continuously develop and deploy new supported PVMs, and thereby new execution environments, via protocol amendments. In our upcoming protocol proposal, we plan to introduce a first <span class="caps">PVM</span> supporting <a href="https://webassembly.org/">WebAssembly</a> (<span class="caps">WASM</span>) — although only available via bleeding-edge test networks.</p>
<p>WebAssembly-based rollups come with a native token bridge based on <a href="https://www.marigold.dev/post/tickets-for-dummies">Tickets</a>, similar to the one used for TORUs, and with a bare-bones execution environment supporting token exchanges between Layer 1 and Layer 2, gas monitoring, and self-governance. On top of this infrastructure, users can write custom rollup kernels, ranging from application-specific rollups to full-scale <em>“parachains”</em>.</p>
<p>Finally, a proof-of-concept rollup kernel will be also provided with the proposal: a token exchange with <span class="caps">BLS</span> signature verification. We aim to integrate this with the in-development <em>Data Availability Layer</em> to demonstrate scaling up to higher throughput values.</p>
<!—
As we hinted in an earlier [scalability preview](https://research-development.nomadic-labs.com/tezos-is-scaling.html), Smart Contract Optimistic Rollups (aka SCORUs) provide a generic infrastructure to integrate arbitrary Layer 2 chains in Tezos, and they can be seen as a natural evolution of [TORUs](https://research-development.nomadic-labs.com/toru-introduction.html). SCORUs not only provide the same properties TORUs do — they are _permissionless_, they minimize trust, and they are moreover _capital efficient_ — but they are also more expressive. Indeed, a SCORU can implement any computational device, as long as its semantics can be described as a Proof-producing Virtual Machine (PVM) in the Tezos protocol.
SCORUs get this extra expressiveness thanks to _interactive_ refutation proofs: TORUs use a single Layer 1 operation to refute a wrong commitment, and thus limit the amount of computations represented by a single commitment — due to the size constraint imposed by having to fit the whole refutation proof a single Layer 1 operation. In contrast, SCORUs allows the _committer_ and the _refuter_ to play an interactive refutation game about an arbitrary long execution using several Layer 1 operations. Thanks to this technique, the amount of operations per second handled by a SCORU is only limited by the bandwidth of the Tezos Layer 1 — which makes SCORUs a more modular scaling solution for the Tezos blockchain.
The genericity of the PVM infrastructure enables a continuous development and deployment of supported PVMs via protocol amendments, paving the way for the integration of any smart-contract execution environment in the Tezos blockchain. In our upcoming protocol proposal, we plan to introduce a first PVM supporting [WebAssembly](https://webassembly.org/), (WASM) — although only available via bleeding-edge test networks.
### A WASM PVM for Tezos
We extend the underlying SCORU base protocol logic to let users originate optimistic rollups running a deterministic subset of WebAssembly. As in TORUs, WebAssembly-based rollups come with a native token bridge based on [Tickets](https://www.marigold.dev/post/tickets-for-dummies), suitable for encoding both fungible and non-fungible assets.
We look forward to providing with the protocol proposal a bare-bones execution environment supporting token exchanges between Layer 1 and Layer 2, gas monitoring, and self-governance. On top of this infrastructure, users can write custom rollup kernels, ranging from application-specific rollups to full-scale _”parachains”_.
Moreover, Kathmandu will provide safe Rust bindings to the low-level WebAssembly host capabilities, providing end users the tools to develop safe and performant rollup kernels. —->
<h3 id="start-building-with-scorus-on-mondaynet"><a class="toclink" href="#start-building-with-scorus-on-mondaynet">Start building with SCORUs on Mondaynet</a></h3>
<p>Starting with the Kathmandu protocol proposal, we want to take a new approach to the protocol development process. Taking in feedback from the recent protocol proposal periods, Kathmandu marks a move towards a <em>“continuous development”</em> process that reduces friction and enables ecosystem developers to build support for new features incrementally in close collaboration with the core protocol developers.</p>
<p>We observe that the current development rhythm does not always provide sufficient time for infrastructure and application developers to embrace and support the new protocol features, including experimenting with them on test networks.</p>
<p>We are therefore changing the process of integrating major new features into Tezos protocol proposals so as to:</p>
<ul>
<li>make sure that together with ecosystems builders, we can provide end-to-end support for new features of the Tezos protocol;</li>
<li>enable the community to provide valuable feedback to protocol developers before features are locked by the governance process;</li>
<li>increase the chance of discovering bugs before proposals are injected and avoiding re-injections and protocol overrides.</li>
</ul>
<p>The <span class="caps">SCORU</span> core business logic and the Wasm <span class="caps">PVM</span> are in a mature enough state for the community to start developing and testing applications and infrastructure on testnets. But, given the new approach outlined above, we want to provide the community with extra time to develop and test infrastructure and Layer 2 applications. As a result, SCORUs will not be enabled on Tezos Mainnet by the Kathmandu protocol proposal.</p>
<p>Given that the “Kathmandunet” test network is meant to mirror the protocol proposal subject to on-chain governance, it will not be spawned with support for SCORUs enabled either. Instead, we encourage the community to start building and testing support for SCORUs on the bleeding edge <a href="https://teztnets.xyz/mondaynet-about">Mondaynet</a> and <a href="https://teztnets.xyz/dailynet-about">Dailynet</a> testnets.</p>
<h2 id="whats-coming-in-kathmandu"><a class="toclink" href="#whats-coming-in-kathmandu">What’s coming in Kathmandu</a></h2>
<p>In the following, we focus on the features we are confident will be enabled in the upcoming Kathmandu protocol proposal.</p>
<h3 id="first-steps-towards-pipelined-block-validation"><a class="toclink" href="#first-steps-towards-pipelined-block-validation">First steps towards pipelined block validation</a></h3>
<p><a href="https://gitlab.com/tezos/tezos/-/milestones/23#tab-issues">The validation Pipelining project</a> aims to streamline the block validation process, in order to reduce the number of times they need to be applied (that is, executed) across the Tezos Layer 1 network. It also aims to propagate blocks <em>before</em> fully applying them, but <em>after</em> having fastly <em>validated</em> a set of preconditions on the block and its operation payload which guarantees their correctness. These goals combine to pursue increasing throughput for Tezos’ Layer 1, without compromising the network’s safety.</p>
<p>The Kathmandu protocol proposal will implement first steps in this direction by focusing on disentangling validation and application for <em>manager operations</em><sup id="fnref:manager"><a class="footnote-ref" href="#fn:manager">1</a></sup> inside the Tezos protocol. That is, we separate the business logic of addressing the question of whether an operation can be safely included in a block, from its actual inclusion in a block.</p>
<!--
We build upon earlier work on the [manager operations precheck plugin](https://tezos.gitlab.io/jakarta/precheck.html), introduced with [Ithaca](https://research-development.nomadic-labs.com/announcing-tezos-9th-protocol-upgrade-proposal-ithaca.html#precheck-of-operations), which implemented this business logic in an _ad hoc_ manner. For Kathmandu, we cement this business logic inside the Tezos protocol, by refactoring and isolating the implementation of both parts into separate modules exposing sound and efficient validation primitives. -->
<p>The second part of the project is planned for future protocol proposals, and it will consist of integrating the remaining parts necessary for a complete pipelined validation of blocks and operations to be fully operational inside the Tezos protocol.</p>
<h3 id="improved-randomness-using-verifiable-delay-functions"><a class="toclink" href="#improved-randomness-using-verifiable-delay-functions">Improved randomness using Verifiable Delay Functions</a></h3>
<!—
All previous Tezos protocols up to now relied on a simple RANDAO scheme to generate a random seed at each cycle, which in turn feeds a PRNG. This is particularly important for the attribution of rights (and rewards) related to consensus.
RANDAO however is susceptible to a slight malleability due to the fact that computing a hash function is extremely fast, allowing the last revealer to foresee what is going to be next seed and the
corresponding rights.
Verifiable Delay Functions (VDF) are novel cryptographic primitives that allow a verifier to check that a value was computed by a prover as a result of a long computation.
Appending a VDF phase after our current RANDAO scheme, prevents any participant from anticipating what could be the next seed because the fast hash function is replaced by a (provably) long computation.
All Tezos protocols versions so far — including Jakarta — rely on a simple [RANDAO-like](https://tezos.gitlab.io/jakarta/randomness_generation.html) scheme to generate a random seed at each cycle, which in turn feeds a Pseudo-Random Number Generator (PRNG). Sound randomness is paramount for enabling the Tezos protocol to fairly decide the attribution of [consensus rights](https://tezos.gitlab.io/jakarta/consensus.html#validator-selection-staking-balance-active-stake-and-frozen-deposits), which ultimately affect [participation rewards](https://tezos.gitlab.io/jakarta/consensus.html#rewards).
This scheme is however susceptible to a slight malleability, if computing the new seed (by hashing a previous seed and known revealed _nonces_) can be performed fastly enough. The last _revealer_ could then foresee what the immediate next seed would be, and then decide to reveal or not depending on which of the resulting seeds allocates them better rights. Even if The Tezos protocol does enforce economic incentives to discourage this attack[^slash], we propose to improve confidence in the Tezos protocol’s distributed randomness generation mechanism with the use of [Verifiable Delay Functions](https://tezos.gitlab.io/alpha/randomness_generation.html#verifiable-delay-function) (VDFs).
[^slash]: A delegate which fails to reveal a nonce within the cycle will loose the endorsement rewards (part of the consensus rewards) for that cycle — this penalty outweighs the potential gains of an attacker.
VDFs are novel cryptographic primitives that allow a _verifier_ to quickly assert that a value — “a proof” — was computed by a _prover_, as a result of an expensive computation. That is, VDFs proofs can be verified _quickly_, but producing a correct proof takes significantly longer time. Appending a VDF phase after our current RANDAO scheme prevents participants from anticipating what the next seed could be, because it would not be able produce the VDF proof within the expected time-frame.
—>
<p>All Tezos protocols versions so far — including Jakarta — rely on a simple <a href="https://tezos.gitlab.io/alpha/randomness_generation.html"><span class="caps">RANDAO</span>-like</a> scheme to generate a random seed at each cycle, which in turn feeds a Pseudo-Random Number Generator (<span class="caps">PRNG</span>). Sound randomness is paramount for enabling the Tezos protocol to fairly decide the attribution of <a href="https://tezos.gitlab.io/jakarta/consensus.html#validator-selection-staking-balance-active-stake-and-frozen-deposits">consensus rights</a>, which ultimately affect <a href="https://tezos.gitlab.io/jakarta/consensus.html#rewards">participation rewards</a>.</p>
<p>In order to strengthen the Tezos protocol’s distributed randomness generation mechanism, we propose integrating <a href="https://tezos.gitlab.io/alpha/randomness_generation.html#verifiable-delay-function">Verifiable Delay Functions</a> (VDFs). These are novel cryptographic primitives that allow a <em>verifier</em> to quickly assert that a value was computed as a result of an expensive computation. Appending a <span class="caps">VDF</span> phase after our current <span class="caps">RANDAO</span> scheme adds an extra layer of security to the protocol’s random seeds.</p>
<h3 id="event-logging-in-tezos-smart-contracts"><a class="toclink" href="#event-logging-in-tezos-smart-contracts">Event logging in Tezos smart contracts</a></h3>
<p>The Kathmandu protocol proposal will provide support for Tezos smart contracts to cheaply emit <a href="https://tezos.gitlab.io/alpha/event.html?highlight=event">on-chain events</a> via statically typed event data attachments. This new feature will enable DApp developers to send publicly visible on-chain messages in order to trigger effects in off-chain applications. For instance, events can be notifications of changes in contract states, be it explicit or implicit. We aim to provide a uniform interface with existing token transfer facilities to allow event logging without origination of any extra contracts.</p>
<p>Further details on the motivation behind this feature can be found in <a href="https://tezos.gitlab.io/alpha/event.html">Contract event logging</a>.</p>
<h3 id="support-for-permanent-test-networks"><a class="toclink" href="#support-for-permanent-test-networks">Support for permanent test networks</a></h3>
<p>The Kathmandu protocol proposal will also include a new testnet-specific governance mechanism necessary to support a permanent test network, <a href="https://forum.tezosagora.org/t/turning-ithacanet-into-a-permanent-testnet-ghostnet">Ghostnet</a>.</p>
<!--
The Kathmandu protocol proposal will also include a new testnet-specific governance mechanism necessary to enable forking [Ithacanet](https://tezos.gitlab.io/introduction/test_networks.html#ithacanet), the test network running the protocol currently active on Tezos Mainnet, into a permanent test network, [Ghostnet](https://forum.tezosagora.org/t/turning-ithacanet-into-a-permanent-testnet-ghostnet).
--->
<p>For any chain other than Tezos Mainnet, this mechanism lets the chain originator designate a special account, that can upgrade the protocol unilaterally with a new protocol proposal. Instead of going through the normal governance election cycles, the chain will upgrade straight away to the new protocol. This will allow for the creation of a long-running testnet, Ghostnet, where protocol upgrades will be centrally managed by Oxhead Alpha and which will closely follow Tezos mainnet upgrades.</p>
<p>Further details are available in the <a href="https://gitlab.com/tezos/tzip/-/merge_requests/178"><span class="caps">TZIP</span></a> advocating for this feature. The implementation is a contribution of <a href="https://gitlab.com/g.b.fefe">G.-B. Fefe</a>, a community member not affiliated with the core developing teams behind this protocol proposal.</p>
<h2 id="looking-ahead"><a class="toclink" href="#looking-ahead">Looking ahead</a></h2>
<p>The contents of the Kathmandu proposal will be finalized shortly, after the last stages of stabilization are completed. As customary with recent proposals, a dedicated test network Kathmandunet will be spawned shortly after the protocol is announced. Moreover, and <a href="#Start-building-with-SCORUs-on-Mondaynet">as we mentioned above</a>, we invite the community to start testing the upcoming <span class="caps">SCORU</span> functionality in the <a href="https://teztnets.xyz/mondaynet-about">Mondaynet</a> test network, and we will provide more details shortly.</p>
<p>We set out to make 2022 a year of ground-breaking advances in the Tezos technology stack, and we believe Kathmandu follows naturally the path set out with <a href="https://research-development.nomadic-labs.com/ithaca-is-live.html">Ithaca</a> and <a href="https://research-development.nomadic-labs.com/announcing-jakarta-two.html">Jakarta</a> protocol upgrades. We look forward to sharing these new exciting features with you all!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:manager">
<p><em>Manager operations</em> include transfers, smart contract originations, calls to smart contracts, etc. In short, any <em>fee-paying</em> operation in competition for block space. Manager operations have a <em>counter</em> associated (unsurprisingly) to their manager, which is the Tezos account which signs the operation and which pays the fees. <a class="footnote-backref" href="#fnref:manager" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Announcing Tezt2022-06-16T14:00:00+02:002022-06-16T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-06-16:/announcing-tezt.html<p>Tezt is a test framework for OCaml that has been developed and used by Nomadic Labs</p><p>Tezt (pronounced <a href="http://ipa-reader.xyz/?text=t%C9%9Bzti">/tɛzti/</a>) is
a test framework for OCaml that has been developed and used at Nomadic
Labs <a href="https://research-development.nomadic-labs.com/announcing-octez.html">to test
Octez</a>,
an OCaml implementation of the Tezos blockchain. It has become quite
mature and we feel it would benefit the OCaml community at large, so
we are releasing it publicly as a standalone product.</p>
<h2 id="use-cases"><a class="toclink" href="#use-cases">Use Cases</a></h2>
<p>Tezt is well suited for:</p>
<ul>
<li>unit tests;</li>
<li>integration tests;</li>
<li>regression tests.</li>
</ul>
<p>This allows you to seamlessly use the same framework for most of your tests.</p>
<h2 id="features"><a class="toclink" href="#features">Features</a></h2>
<p>General features:</p>
<ul>
<li>easy and flexible test selection from the command-line;</li>
<li>run tests in parallel in separate processes;</li>
<li>automatically split tests into well-balanced batches for running in separate <span class="caps">CI</span> jobs;</li>
<li>various reporting options, including JUnit for <span class="caps">CI</span> integration;</li>
<li>write assertions (equalities, inequalities, existence in lists,
<em>etc.</em>) that result in nice error messages when they fail.</li>
</ul>
<p>For integration tests:</p>
<ul>
<li>run external processes, automatically logging their output and
terminating them at the end of a test;</li>
<li>declare temporary files that are automatically removed at the end
of a test;</li>
<li>external processes can be run on distant runners through <span class="caps">SSH</span>;</li>
<li>a <span class="caps">JSON</span> module is provided to test <span class="caps">REST</span> APIs.</li>
</ul>
<p>For regression tests:</p>
<ul>
<li>capture some strings (e.g. output from external processes);</li>
<li>apply regular-expression-based substitutions to make those outputs deterministic;</li>
<li>compare with previous captured outputs.</li>
</ul>
<p>Additionally, substantial effort was put into the user experience. Many small details that may not seem like much individually, but add up to significantly improve everyday use. These include:</p>
<ul>
<li>colorful logs: for instance, each external process gets its own color;</li>
<li>by default, logs are quiet until an error occurs, in which case
previous log messages are displayed to get some context;</li>
<li>an option allows you to see all commands that are used to run
external processes, making it easy for you to copy-paste them and reproduce;</li>
<li>if one of your tests is flaky (i.e. randomly fails), an option
allows you to run it in a loop to more easily reproduce the error;</li>
<li><span class="caps">JSON</span> error messages show the location of ill-typed <span class="caps">JSON</span> value;</li>
<li>all functions are clearly documented.</li>
</ul>
<h2 id="a-simple-test"><a class="toclink" href="#a-simple-test">A Simple Test</a></h2>
<p>Create a <code>dune</code> file containing:</p>
<div class="highlight"><pre><span></span><code>(test (name main) (libraries tezt))
</code></pre></div>
<p>Now create a file named <code>main.ml</code>. To add a test, register it with <code>Test.register</code>:</p>
<div class="highlight"><pre><span></span><code><span class="k">open</span> <span class="nc">Tezt</span>
<span class="k">open</span> <span class="nn">Tezt</span><span class="p">.</span><span class="nc">Base</span>
<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span>
<span class="nn">Test</span><span class="p">.</span><span class="n">register</span> <span class="o">~__</span><span class="n">FILE__</span>
<span class="o">~</span><span class="n">title</span><span class="o">:</span> <span class="s2">"example test"</span>
<span class="o">~</span><span class="n">tags</span><span class="o">:</span> <span class="o">[</span><span class="s2">"example"</span><span class="o">;</span> <span class="s2">"addition"</span><span class="o">]</span>
<span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-></span>
<span class="k">if</span> <span class="mi">1</span> <span class="o">+</span> <span class="mi">1</span> <span class="o"><></span> <span class="mi">2</span> <span class="k">then</span> <span class="nn">Test</span><span class="p">.</span><span class="n">fail</span> <span class="s2">"1 + 1 is not 2"</span><span class="o">;</span>
<span class="kt">unit</span>
<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span>
<span class="c">(* Call this after you are done registering tests. *)</span>
<span class="nn">Test</span><span class="p">.</span><span class="n">run</span> <span class="bp">()</span>
</code></pre></div>
<p>Let’s see what is going on here.</p>
<ul>
<li><code>~__FILE__</code> tells Tezt which source file is defining this test. You
can tell Tezt to run all tests that are registered in a given
source file with <code>--file</code> (or <code>-f</code>). Here, that would be <code>--file
main.ml</code>.</li>
<li><code>~title</code> should be a short descriptive title for your test. You can
tell Tezt to run this particular test with <code>--test 'example test'</code>.</li>
<li><code>~tags</code> is a list of identifiers that can be used to select a
subset of your tests from the command-line. For instance, you can
run all tests that have tag <code>addition</code> simply by adding <code>addition</code>
on the command-line. You can also specify negative tags; for
instance, you can have a tag <code>slow</code> for tests that take longer to
run, and exclude them with <code>/slow</code>.</li>
<li>The last argument is a function that implements the test.</li>
<li><code>Test.fail</code> raises an exception, similar to <code>Failure</code>. It stops the
test. Tezt then cleans up if there is something to clean, and
reports the error.</li>
<li><code>unit</code> is <code>Lwt.return_unit</code>. All tests are written in the Lwt monad.</li>
<li><code>Test.run</code> tells Tezt that all tests have been registered. Tezt
takes it from here, handling command-line options and running tests
that it should run.</li>
</ul>
<p>To run your test, run either:</p>
<div class="highlight"><pre><span></span><code>dune runtest
</code></pre></div>
<p>or:</p>
<div class="highlight"><pre><span></span><code><span class="nv">dune</span><span class="w"> </span><span class="k">exec</span><span class="w"> </span><span class="nv">main</span>.<span class="nv">exe</span>
</code></pre></div>
<p>You can get a list of command-line options with:</p>
<div class="highlight"><pre><span></span><code><span class="nv">dune</span><span class="w"> </span><span class="k">exec</span><span class="w"> </span><span class="nv">main</span>.<span class="nv">exe</span><span class="w"> </span><span class="o">--</span><span class="w"> </span><span class="o">--</span><span class="nv">help</span>
</code></pre></div>
<p>In particular, you can try the <code>--list</code> option, which gives you the
list of registered tests and their tags. Don’t forget to use <code>--</code> to
separate Dune arguments and Tezt arguments:</p>
<div class="highlight"><pre><span></span><code><span class="nv">dune</span><span class="w"> </span><span class="k">exec</span><span class="w"> </span><span class="nv">main</span>.<span class="nv">exe</span><span class="w"> </span><span class="o">--</span><span class="w"> </span><span class="o">--</span><span class="nv">list</span>
</code></pre></div>
<p>Some basic options that you may be interested in are:</p>
<ul>
<li><code>--list</code> to get the list of registered tests;</li>
<li><code>--test TITLE</code> (or <code>-t TITLE</code>) to select a test from its title;</li>
<li><code>--file FILE.ml</code> (or <code>-f FILE.ml</code>) to select tests from their
source file;</li>
<li><code>--verbose</code> (or <code>-v</code>), <code>--info</code> (or <code>-i</code>) to control log verbosity;</li>
<li><code>--log-file FILE</code> to store verbose logs in <span class="caps">FILE</span>;</li>
<li><code>--keep-going</code> (or <code>-k</code>) to continue with remaining tests even if a
test fails;</li>
<li><code>--jobs N</code> (or <code>-j N</code>) to run up to N tests in parallel.</li>
</ul>
<h2 id="a-simple-integration-test"><a class="toclink" href="#a-simple-integration-test">A Simple Integration Test</a></h2>
<p>An integration test is just a test that runs external processes. It
often also manipulates temporary files. Here is a simple example:</p>
<div class="highlight"><pre><span></span><code><span class="k">open</span> <span class="nc">Tezt</span>
<span class="k">open</span> <span class="nn">Tezt</span><span class="p">.</span><span class="nc">Base</span>
<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span>
<span class="nn">Test</span><span class="p">.</span><span class="n">register</span> <span class="o">~__</span><span class="n">FILE__</span>
<span class="o">~</span><span class="n">title</span><span class="o">:</span> <span class="s2">"example integration test"</span>
<span class="o">~</span><span class="n">tags</span><span class="o">:</span> <span class="o">[</span><span class="s2">"example"</span><span class="o">;</span> <span class="s2">"cat"</span><span class="o">]</span>
<span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-></span>
<span class="k">let</span> <span class="n">filename</span> <span class="o">=</span> <span class="nn">Temp</span><span class="p">.</span><span class="n">file</span> <span class="s2">"test.txt"</span> <span class="k">in</span>
<span class="n">with_open_out</span> <span class="n">filename</span> <span class="o">(</span><span class="k">fun</span> <span class="n">ch</span> <span class="o">-></span>
<span class="n">output_string</span> <span class="n">ch</span> <span class="s2">"test"</span> <span class="o">^</span> <span class="n">string_of_int</span> <span class="o">(</span><span class="nn">Random</span><span class="p">.</span><span class="n">int</span> <span class="mi">1000</span><span class="o">));</span>
<span class="k">let</span><span class="o">*</span> <span class="n">output</span> <span class="o">=</span> <span class="nn">Process</span><span class="p">.</span><span class="n">run_and_read_stdout</span> <span class="s2">"cat"</span> <span class="o">[</span> <span class="n">filename</span> <span class="o">]</span> <span class="k">in</span>
<span class="k">if</span> <span class="n">output</span> <span class="o">=~!</span> <span class="n">rex</span> <span class="s2">"^test</span><span class="se">\\</span><span class="s2">d{1,4}$"</span> <span class="k">then</span>
<span class="nn">Test</span><span class="p">.</span><span class="n">fail</span> <span class="s2">"got %S instead of the expected output"</span> <span class="n">output</span><span class="o">;</span>
<span class="kt">unit</span>
<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Test</span><span class="p">.</span><span class="n">run</span> <span class="bp">()</span>
</code></pre></div>
<p>Let’s see what is going on here. This tests uses three functions from
the <code>Base</code> module of Tezt:</p>
<ul>
<li>
<p><code>Temp.file "test.txt"</code> returns a filename of the form
<code>/tmp/tezt-1234/1/test.txt</code>. This file will be automatically
removed at the end of the test unless you use <code>--keep-temp</code>, or
<code>--delete-temp-if-success</code> and the test fails.</p>
</li>
<li>
<p><code>with_open_out</code> opens a file for writing, gives you the
<code>out_channel</code>, and ensures the file is closed after.</p>
</li>
<li>
<p><code>let*</code> is <code>Lwt.bind</code>.</p>
</li>
</ul>
<p>Then it uses the <code>Process</code> module to execute <code>cat</code> on the temporary
file and check its output. The <code>Process</code> module will log all output
from <code>cat</code> in verbose mode. This is particularly convenient for debugging.</p>
<p>Finally, this test uses <code>=~!</code>, which means: check that the
left-hand-side does not match the regular expression at the
right-hand-side. Regular expressions are often convenient to use with
integration tests, so some easy-to-use regular expression operators
are provided by <code>Tezt.Base</code>.</p>
<p>You can run this test like any other test. One option that may be
useful is <code>--commands</code> (or <code>-c</code>): it tells Tezt to print
commands that it runs. It quotes them using shell syntax so that you
can copy-paste them into your terminal to easily reproduce.</p>
<h2 id="a-simple-regression-test"><a class="toclink" href="#a-simple-regression-test">A Simple Regression Test</a></h2>
<p>Regression tests are tests that produce a string which should never
change. This string is stored in a file that you can commit in your
repository. When you run your test, the string that it produces will
be compared with the contents of this file, and the test will fail if
it differs.</p>
<p>Regression tests are in particular useful to check that the output of
external processes stay unchanged, so they are often used in
integration tests. But they can also be used for unit tests.</p>
<p>Here is an example regression test:</p>
<div class="highlight"><pre><span></span><code><span class="k">open</span> <span class="nc">Tezt</span>
<span class="k">open</span> <span class="nn">Tezt</span><span class="p">.</span><span class="nc">Base</span>
<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span>
<span class="nn">Regression</span><span class="p">.</span><span class="n">register_test</span> <span class="o">~__</span><span class="n">FILE__</span>
<span class="o">~</span><span class="n">title</span><span class="o">:</span> <span class="s2">"example regression test"</span>
<span class="o">~</span><span class="n">tags</span><span class="o">:</span> <span class="o">[</span><span class="s2">"example"</span><span class="o">]</span>
<span class="o">~</span><span class="n">output_file</span><span class="o">:</span> <span class="s2">"example.txt"</span>
<span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-></span>
<span class="k">let</span><span class="o">*</span> <span class="n">output</span> <span class="o">=</span>
<span class="nn">Process</span><span class="p">.</span><span class="n">run</span> <span class="o">~</span><span class="n">hooks</span><span class="o">:</span> <span class="nn">Regression</span><span class="p">.</span><span class="n">hooks</span>
<span class="s2">"git"</span> <span class="o">[</span> <span class="s2">"--help"</span> <span class="o">]</span>
<span class="k">in</span>
<span class="nn">Regression</span><span class="p">.</span><span class="n">capture</span> <span class="o">(</span><span class="n">string_of_int</span> <span class="o">(</span><span class="mi">1</span> <span class="o">+</span> <span class="mi">1</span><span class="o">));</span>
<span class="kt">unit</span>
<span class="k">let</span> <span class="bp">()</span> <span class="o">=</span> <span class="nn">Test</span><span class="p">.</span><span class="n">run</span> <span class="bp">()</span>
</code></pre></div>
<p>Let’s see what is going on here.</p>
<ul>
<li>This test is registered with <code>Regression.register_test</code> instead of
<code>Test.register</code>. This tells the test that it shall produce a file
with the captured regression output.</li>
<li><code>output_file</code> is the name of the file to produce. It is prefixed
with <code>tezt/_regressions/</code>, so here the output file is
<code>tezt/_regressions/example.txt</code>. You can override this prefix with
<code>--regression-dir</code>.</li>
<li>We use <code>~hooks</code> to tell the <code>Process</code> module to capture the output
of the process in the regression output file.</li>
<li>We also manually capture the result of a computation.</li>
</ul>
<p>You need to run the test once with <code>--reset-regressions</code> to generate the output file:</p>
<div class="highlight"><pre><span></span><code><span class="n">dune</span><span class="w"> </span><span class="n">exec</span><span class="w"> </span><span class="n">main</span><span class="p">.</span><span class="n">exe</span><span class="w"> </span><span class="o">--</span><span class="w"> </span><span class="n">regression</span><span class="w"> </span><span class="o">--</span><span class="n">reset</span><span class="o">-</span><span class="n">regressions</span>
</code></pre></div>
<p>Here we also specify the <code>regression</code> tag on the command-line. This is optional but this tells Tezt to only run tests which have tag <code>regression</code>. This tag is automatically added by <code>Regression.register_test</code>.</p>
<p>Next time you run your tests, for instance with <code>dune exec main.exe</code>,
the output of the test will be compared with
<code>tezt/_regressions/example.txt</code>. If it differs, the test will fail.</p>
<p>Regression tests are convenient because they are fast to
write. However, they do have some drawbacks:</p>
<ul>
<li>
<p>the expected output is not in the source code of the test itself
(an alternative for this is to use <code>ppx_expect</code>);</p>
</li>
<li>
<p>if the output contains non-deterministic values, such as time data
or random values, you first need to replace them with fixed values
(this can be done with custom <code>~hooks</code>).</p>
</li>
</ul>
<h2 id="ci-integration"><a class="toclink" href="#ci-integration"><span class="caps">CI</span> Integration</a></h2>
<p>Our tests run in GitLab’s <span class="caps">CI</span> for all merge requests. This prompted us
to make Tezt well integrated with the <span class="caps">CI</span>. Here are some of the
features that were built with that in mind.</p>
<ul>
<li>
<p>GitLab’s job interface has a limit for how many lines of output can
be shown for a given job. By default Tezt only shows detailed logs
in case of a failure, and only the most recent lines (the exact
amount is a command-line option). This means that we never reach
GitLab’s limit. We can still see the full logs if we want: we use
the <code>--log-file</code> option and store the log file as an artifact.</p>
</li>
<li>
<p>Tezt can generate record files that store the time each test took
to run. It can then use those records to automatically split tests
into a partition where each subset takes roughly the same amount of
time. And you can tell Tezt to run only one of this subset in each
<span class="caps">CI</span> job. The result is that you can easily split your tests in
well-balanced parallel jobs in the <span class="caps">CI</span>.</p>
</li>
<li>
<p>Tezt can generate JUnit reports. JUnit is supported by GitLab. This
gives the ability to show a summary of test results in the merge
request interface.</p>
</li>
<li>
<p>Because Tezt can run external processes using <span class="caps">SSH</span>, you could have
the <span class="caps">CI</span> runner be the puppetmaster for a bigger cluster of
machines. We don’t actually do that, but we could.</p>
</li>
</ul>
<p>Note that outside of the JUnit format, none of these features assume
that your <span class="caps">CI</span> runs on GitLab, so they should port rather easily to
other CIs.</p>
<h2 id="and-more"><a class="toclink" href="#and-more">And More</a></h2>
<p>Tezt has a rather long list of command-line options to configure its
behavior. Run your executable with <code>--help</code> to see it, or read Tezt’s
<code>cli.ml</code> source file. We already had a look at some of these options
in section A Basic Test and we already mentioned some others. But
there are too many to list here, as Tezt, being heavily used, has
aggregated many small yet useful features over the time.</p>
<h2 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h2>
<p>Tezt development started more than two years ago. At the time, we had
two frameworks for integration tests. Flextesa, which was written in
OCaml but failed to convince all developers (possibly because it
focused on interactive tests), and a Python-based framework. Most
integration tests were thus written in Python. As primarily OCaml
developers, we were eager to write our tests in OCaml.</p>
<p>Tezt thus began with a focus on integration tests and was developed
with user experience, simplicity of implementation and ease of use in
mind. It quickly convinced a few early adopters at Nomadic Labs, and
expanded from there. Now, no new test is written in Python, all new
integration tests are written using Tezt.</p>
<p>We eventually realized that Tezt could also be used for unit tests. We
did not actually transition to Tezt for unit tests though; most unit
tests are written using Alcotest, a test library for OCaml. But some
developers are starting to voice preference for Tezt for unit testing,
either because they like its focus on user experience, or because they
like its <span class="caps">CI</span> integration capabilities like auto-balancing, or just
because they would prefer to only have one framework. Tezt provides
most of Alcotest’s features, except (for now) integration with QCheck,
a library for property-based testing. It should, however, not be very
hard to integrate QCheck with Tezt, and we may decide to make the jump
one day.</p>
<p>It’s now clear to us that Tezt is a success with Octez developers, and we
see no reason to keep it for ourselves. Version 1.0.0 was already released on
opam a while ago, but with no announcement — it was mainly so that we could use
it ourselves on other Tezos-related projects. We just released version 2.0.0
on opam: run <code>opam install tezt</code> and start tezting!</p>
<p>You can find the <span class="caps">API</span> documentation <a href="https://tezos.gitlab.io/api/odoc/_html/tezt/Tezt/index.html">here.</a></p>Transaction Optimistic Rollups – a stepping stone for Tezos2022-06-09T16:00:00+02:002022-06-09T16:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-06-09:/toru-introduction.html<p>The hows and whys of the first, experimental implementation of optimistic rollups in the Tezos protocol.</p><p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: Transaction optimistic rollups are an experimental solution meant for short-term scaling and for adapting ecosystem infrastructure to work with rollups, while smart contracts rollups are under development.</strong></p>
<p>With the <a href="https://research-development.nomadic-labs.com/announcing-tezos-10th-protocol-upgrade-proposal-jakarta.html">Jakarta protocol proposal</a>, the first step on the road towards scaling Tezos with optimistic rollups has been taken.</p>
<p>To quickly recap: Optimistic rollups enable higher throughput by moving the computationally demanding validation of transactions away from the main chain, <em>Layer 1</em>, to dedicated nodes, <em>Layer 2</em>.</p>
<p>This approach is far from new, but optimistic rollups do so while delegating consensus – the authority to determine the true state of the Layer 2 ledger – to Layer 1.</p>
<p>All data necessary to verify correctness of rollup activity remains available on Layer 1, along with an on-chain dispute mechanism for automatically correcting dishonest behavior by rollup nodes.</p>
<p>This way, optimistic rollups enable the use of powerful, centralized systems for validation of Layer 2 activity, while integrity is ultimately ensured by the security and censorship resistance of the Layer 1 protocol that hosts the rollup.</p>
<p>For a broader understanding of optimistic rollups and the rationale behind them as a scaling solution, we recommend our recent <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html">blog post</a> on the topic.</p>
<h2 id="torus-as-a-stepping-stone"><a class="toclink" href="#torus-as-a-stepping-stone">TORUs as a stepping stone</a></h2>
<p><a href="https://tezos.gitlab.io/jakarta/transaction_rollups.html">Transaction Optimistic Rollups</a>, TORUs, are the first implementation of optimistic rollups in the Tezos protocol. As the name implies, they allow for exchanges of assets, but not execution of smart contracts.</p>
<p>The purpose of TORUs is to</p>
<ul>
<li>Demonstrate Tezos’ ability to implement scaling solutions through protocol upgrades</li>
<li>Provide a first implementation of optimistic rollups for ecosystem developers to start experimenting with</li>
<li>Pave the way for <em>smart contract optimistic rollups</em> – a more advanced implementation, expected to be part of a protocol proposal later in 2022.</li>
</ul>
<p>TORUs should be treated as a short-term and experimental scaling solution. For this reason, they are introduced with a sunset of 1 year. This can be changed in future protocol upgrades, should the community desire it.</p>
<h2 id="a-roll-of-tickets"><a class="toclink" href="#a-roll-of-tickets">A roll of tickets</a></h2>
<p>A transaction optimistic rollup is an entity with its own address residing on the Tezos main chain.</p>
<p>The rollup holds assets that are deposited from – and that can be withdrawn back to – the main chain, Layer 1. These assets can be transferred across rollup-native Layer 2 addresses at a higher throughput than what is possible on Layer 1.</p>
<p>Deposited assets are represented as <em><a href="https://tezos.gitlab.io/active/michelson.html#michelsontickets">Michelson tickets</a></em> in the rollup. These are a special type of tokens that were introduced in the Edo upgrade in 2021.</p>
<p>With <span class="caps">FA2</span> tokens (Tezos) and <span class="caps">ERC</span>-20 tokens (Ethereum), you don’t actually hold the token in your account. Rather, each token has a centralized smart contract that acts like a bank, keeping track of how much your account holds.</p>
<p>Tickets change this. They are recognized directly by the protocol and can be passed around freely without interacting with a centralized smart contract. If traditional tokens are like a bank account, tickets are like an unforgeable dollar bill. As close to true ownership as it gets.</p>
<p>Besides this desirable property, using tickets has the benefit of simplifying the code when used in TORUs. For a deep dive on Michelson tickets, check out <a href="https://www.marigold.dev/post/tickets-for-dummies">this blog post</a> by our friends at Marigold.</p>
<h2 id="built-to-promote-decentralization"><a class="toclink" href="#built-to-promote-decentralization">Built to promote decentralization</a></h2>
<p>Optimistic rollups on Tezos are implemented as part of the Tezos protocol directly, and not as smart contracts. This is often called <em>enshrined</em> rollups.</p>
<p>An immediate benefit is that it allows for a specialized, more gas- and storage-efficient implementation. For example, deploying a <span class="caps">TORU</span> is way less costly in terms of storage than deploying a smart contract of similar complexity. And as Tezos evolves with further upgrades, new features and optimizations can boost functionality and throughput of all enshrined rollups.</p>
<p>The way TORUs are integrated into the protocol makes them highly decentralized and facilitates broad participation. By design, they are</p>
<ul>
<li><strong>Permissionless:</strong> Anyone can deploy a new <span class="caps">TORU</span>, and anyone can be a node operator for any existing rollup.</li>
<li><strong>Easy to use:</strong> TORUs are identified by specific addresses with the <a href="https://tezos.gitlab.io/jakarta/michelson.html#address-prefixes">prefix</a> <em>txr1</em>. Layer 2 addresses inside the rollup have the prefix <em>tz4</em>. Furthermore, the interface is designed to make interacting with rollups similar to interacting with a smart contract.</li>
<li><strong>Accessible:</strong> It is fairly simple to run a rollup node, which is a (experimental) daemon being developed as part of the Octez code-base. Once configured, it scans for rollup instructions in Layer 1 operations, executes the instructions, and posts a hash representing the updated rollup state to Layer 1.</li>
</ul>
<p>TORUs on Tezos are not run by a single company or group of people. No one “owns” a smart contract containing the <span class="caps">TORU</span>. There’s no admin key. Once deployed, the network “owns” the rollup.</p>
<p>No rules can be applied other than the semantics of included operations. It’s not possible to prevent someone from depositing a ticket or from exchanging this ticket.</p>
<p>Rollup node operators also don’t decide which Layer 2 operations will be processed, or their order of processing, as this is determined by their inclusion in Layer 1 blocks. Any attempt at circumventing this will be neutralized by the Layer 1 <a href="https://tezos.gitlab.io/jakarta/transaction_rollups.html#commitments-and-rejections">dispute mechanism</a>.</p>
<h2 id="using-a-toru"><a class="toclink" href="#using-a-toru">Using a <span class="caps">TORU</span></a></h2>
<p>Interaction with a <span class="caps">TORU</span> happens using the following workflow:</p>
<p><strong>Depositing</strong>: Assets are sent to a Layer 1 smart contract which locks them and mints a corresponding Michelson ticket. The smart contract then sends a transaction containing the ticket and a recipient Layer 2 address to the rollup. It’s worth noting that such ticket minting contracts are not part of the rollup itself and should be seen as a trusted third party. We imagine <span class="caps">NFT</span> marketplaces and <span class="caps">DEX</span>’s integrating this feature, as some trust is already placed in their existing smart contracts by the users.</p>
<p><strong>Transferring:</strong> Just as on Layer 1, a Layer 2 address is tied to a private key held by the owner, which is used to sign transactions. Instructions for the <span class="caps">TORU</span> – signed transactions – are included in Layer 1 operations, but only interpreted by dedicated rollup nodes scanning the blockchain for these.</p>
<p><strong>Withdrawing:</strong> If the recipient of a rollup transfer instruction is a Layer 1 address, the rollup sends the ticket(s) to that address on Layer 1, though it will remain frozen until the dispute period is over. This period will be two weeks initially, but might be changed in the future. Once unfrozen, the ticket can be exchanged for the underlying asset with the smart contract that minted it.</p>
<p>Rollup users who also control a Layer 1 address can of course choose to interact with the rollup by including instructions for their Layer 2 address in Layer 1 operations themselves. However, we expect that most end users will interact with rollups via trusted third-party nodes that batch multiple Layer 2 transactions in a single Layer 1 operation. For example, a wallet provider might run a node that batches all Layer 2 transactions submitted by users of the wallet.</p>
<p>Such batching of transactions is an essential part of the efficiency of rollups. Through the use of <a href="https://www.marigold.dev/post/bls-signatures"><span class="caps">BLS</span> signatures</a>, multiple transaction signatures can also be aggregated into one, which means even more Layer 2 transactions can be included in each Layer 1 block.</p>
<h2 id="why-torus-are-experimental"><a class="toclink" href="#why-torus-are-experimental">Why TORUs are experimental</a></h2>
<p>It bears repeating that TORUs are an experimental solution. They should only be used by projects that are prepared to migrate to a different solution within a year.</p>
<p>The design chosen for TORUs allows for a simpler and faster integration of rollups, but has the side-effect that the maximum throughput of an individual <span class="caps">TORU</span> is capped at a lower level than required for long-term scalability.</p>
<p>A conservative estimate puts throughput for each individual <span class="caps">TORU</span> at 300-500 tps. Total throughput can be increased by running several rollups in parallel, but the design of TORUs means there is a hard limit imposed by Layer 1 blocksize at around 900 tps.</p>
<p>These numbers are however highly dependent on the ecosystem’s use of tools for optimizing rollup capacity, with aggregation of transaction signatures being a major factor. Potential Layer 1 block congestion is another factor.</p>
<p>All in all, TORUs should be seen as a short-term scaling solution and a way for the ecosystem to begin adapting infrastructure to working with rollups, while smart contract optimistic rollups, SCORUs, are being developed.</p>
<p>SCORUs are likely to eventually become the preferred solution for all rollup use cases, as they don’t just add smart contract functionality and other new functionality, but also are far more scalable in terms of throughput.</p>
<p>We will cover SCORUs in much more detail in future blog posts, as their inclusion in a protocol upgrade proposal approaches. Stay tuned!</p>Also coming with Jakarta: spring-cleaning the Michelson interpreter2022-05-12T16:00:00+02:002022-05-12T16:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-05-12:/also-coming-with-jakarta-spring-cleaning-the-michelson-interpreter.html<p>In recent months the Michelson team at Nomadic Labs has launched a
project dedicated to paying off the technical debt in the
interpreter. Michelson is the language of smart contracts in Tezos
blockchain and its interpreter is an integral part of the Tezos
economic protocol. It evolves along with the …</p><p>In recent months the Michelson team at Nomadic Labs has launched a
project dedicated to paying off the technical debt in the
interpreter. Michelson is the language of smart contracts in Tezos
blockchain and its interpreter is an integral part of the Tezos
economic protocol. It evolves along with the protocol and in fact many
new features were added to it in previous upgrades. As happens
commonly when developing software, technical debt accumulated with
these changes and it was decided that it’s time to make a dedicated
effort to pay at least a part of it off.</p>
<h2 id="what-is-technical-debt"><a class="toclink" href="#what-is-technical-debt">What is technical debt?</a></h2>
<p>It is a very broad term encompassing all inefficiencies, suboptimal
design choices, insufficient testing and even bugs that accumulate in
a software project along its lifetime. Writing a good piece of
software is a very difficult and time-consuming task. It relies on
information that isn’t always readily available. Environments change
and so do users’ needs. When pressed by deadlines or uncertainty, developers
often create suboptimal solutions, either erroneously or even
intentionally as a trade-off necessary to deliver on time.</p>
<p>It is called “debt” because allowing for these deficiencies in the short
term allows software to be delivered more quickly, but over time they
take a toll on its quality. Hence it’s
important to keep the technical debt low and pay it
off by fixing inefficiencies on a regular basis.</p>
<h2 id="technical-debt-in-the-michelson-interpreter"><a class="toclink" href="#technical-debt-in-the-michelson-interpreter">Technical debt in the Michelson interpreter</a></h2>
<p>When creating a programming language, an additional complication
with respect to technical debt arises. While the language’s interpreter (or compiler)
accumulates technical debt like any other piece of software,
it must still support all the software previously written in the
language. So it is imperative to ensure that every program
previously written still works in the same way it did before any
upgrade. This requirement makes it challenging to redesign programming
languages and demands extensive testing of each change.</p>
<p>That said, it is sometimes possible (albeit risky) to announce a
breaking change to the language, making it clear to the users that
their programs may no longer work with the new version of the
language. This was done for instance with Python. Python 3 was
incompatible with Python 2 and the breaking changes were at least
partially motivated by technical debt. The transition was so
painful that most Linux distributions still ship with Python 2 (either
as the default or additional interpreter), as a lot of software was
never updated to work with Python 3.</p>
<p>However, in the case of Michelson this approach cannot work, because a
smart contract, once submitted to the blockchain, stays there
forever. The protocol must retain the ability to interpret old
contracts, no matter how many protocol upgrades were implemented since the
contract’s origination. The need to support every existing contract is often
a great obstacle in refactoring and improving the code.</p>
<p>In addition, sometimes certain features of Michelson
turn out to be less useful or safe than expected.
They might also conflict with other features that are later deemed
to be more important. It seems that technical debt can
also arise in utterly normal development processes, because
of external situational changes! For example, users behave differently than
expected, or new scientific developments are announced that would be good
to take advantage of. So, is the blockchain doomed to support those legacy features forever?</p>
<p>Fortunately there is a solution! The Michelson interpreter has a so-called
<strong>legacy mode</strong>, which supports all the features that were ever used
on the blockchain. As the name suggests, it is an optional behaviour,
which is only enabled for executing contracts already existing
on-chain. New contracts before they’re originated are type-checked in
normal mode, which does not have to support all the legacy
features. Indeed, thanks to these two distinct modes of operation it is possible
to deprecate old features and disallow origination of contracts using
them, while still supporting them in old contracts.</p>
<p>But does it really help? The blockchain still has to support deprecated features
in the legacy mode, doesn’t it?</p>
<h2 id="patching-legacy-contracts"><a class="toclink" href="#patching-legacy-contracts">Patching legacy contracts</a></h2>
<p>In the Michelson team at Nomadic Labs, we decided that the time has come
to make steps towards eliminating currently<sup id="fnref:currently-deprecated"><a class="footnote-ref" href="#fn:currently-deprecated">1</a></sup> deprecated
features for good. Contrary to Python, which is a general-purpose language,
Michelson only makes sense within the Tezos blockchain. Because the blockchain
is publicly available, unlike developers of Python, we do have access
to <strong>all</strong> the Michelson programs, or at least to all those that
can have an impact on the blockchain. Although we cannot modify the on-chain
data<sup id="fnref:chain-immutability"><a class="footnote-ref" href="#fn:chain-immutability">2</a></sup>, we can tell the nodes to replace one
contract with another at a later block. Because of the evolving nature
of the Tezos protocol, each new protocol version can (and often does)
alter the way nodes store information about the current state of the
blockchain (called the context). We can use this mechanism to modify contracts stored in
the context. Albeit contracts retain their original form in the blocks
that originated them, they can be patched at a later block by the
protocol migration.</p>
<p>This gives us a clear procedure to remove deprecated features even
from the <strong>legacy mode</strong> of the interpreter:</p>
<ol>
<li>Announce the feature deprecation and remove the feature from the
normal mode.</li>
<li>Wait until the protocol deprecating the feature gets activated.</li>
<li>Type check all the contracts on chain in the normal mode and select
those that fail.</li>
<li>Patch the selected contracts so that they type check successfully again.</li>
<li>Finally remove legacy features from the <strong>legacy mode</strong>.</li>
</ol>
<p>The waiting step is unfortunate, but necessary. We have to wait for
the new protocol with the deprecated feature to get activated, so that
no new contracts are originated using the deprecated feature after we
do the patching.</p>
<p>In the past we did deprecate features in several protocol upgrades,
but this was the first time we actually started patching contracts in
order to remove those features. After some patching and hacking on
the <code>tezos-node</code> we managed to extract all the smart contract scripts
from the mainnet and type checked them using <code>tezos-client typecheck</code>
command. We have found 8 contract scripts that required patching, although
most of them were instantiated multiple times. Patches were mostly
trivial to do and all the scripts type checked successfully again.
But does the story end here? As a careful reader will probably guess,
in fact it has only just begun! We now need to make sure that all the
patched contracts work exactly as they did before.</p>
<p>One example of a deprecated feature we decided to patch away was the
possibility to store typed references to other smart contracts in storage
(of Michelson <code>contract</code> type). These references were at some point forbidden
from appearing in contract storages, so we had to find those contracts that
still held <code>contract</code> references in storage and remove them. The following
patch to smart contract <code>KT1MzfYSbq18fYr4f44aQRoZBQN72BAtiz5</code> is an example of
such a change:</p>
<div class="highlight"><pre><span></span><code><span class="gd">--- patched_contracts/exprtgpMFzTtyg1STJqANLQsjsMXmkf8UuJTuczQh8GPtqfw18x6Lc.original.tz</span>
<span class="gi">+++ patched_contracts/exprtgpMFzTtyg1STJqANLQsjsMXmkf8UuJTuczQh8GPtqfw18x6Lc.patched.tz</span>
<span class="gu">@@ -1,10 +1,5 @@</span>
<span class="w"> </span>{ parameter (or (lambda %do unit (list operation)) (unit %default)) ;
<span class="gd">- storage</span>
<span class="gd">- (pair key_hash</span>
<span class="gd">- (contract</span>
<span class="gd">- (or (option address)</span>
<span class="gd">- (or (pair (option address) (option mutez))</span>
<span class="gd">- (or mutez (or (pair (option address) (option mutez)) address)))))) ;</span>
<span class="gi">+ storage (pair key_hash address) ;</span>
<span class="w"> </span> code { DUP ;
<span class="w"> </span> CAR ;
<span class="w"> </span> IF_LEFT
<span class="gu">@@ -28,6 +23,8 @@</span>
<span class="w"> </span> NIL operation ;
<span class="w"> </span> { DIP { DIP { DUP } ; SWAP } ; SWAP } ;
<span class="w"> </span> { DIP { DIP { DIP { DROP } } } } ;
<span class="gi">+ CONTRACT (or (option address) (or (pair (option address) (option mutez)) (or mutez (or (pair (option address) (option mutez)) address))));</span>
<span class="gi">+ IF_SOME {} {PUSH string "Bad contract in storage"; FAILWITH};</span>
<span class="w"> </span> AMOUNT ;
<span class="w"> </span> SENDER ;
<span class="w"> </span> SOME ;
</code></pre></div>
<p>As can be seen here, <code>contract ...</code> in the storage type is replaced by an
un-typed <code>address</code>. The difference between these values is only in their types:
<code>contract</code> is parametrised by the referenced contract’s type, while <code>address</code>
is not. The latter can be converted into the former, by specifying the said type.
This is exactly what the two lines added in the body of the contract do.</p>
<p>It’s worth noticing that this patched version of the contract can fail at runtime
while the original version couldn’t. It would fail when the stored address points
to a contract of a wrong type. This is unfortunate, but could not be helped. We
will take steps to verify that the change doesn’t break the contract and once we
do it, all should be well.</p>
<h2 id="verifying-the-patched-contracts"><a class="toclink" href="#verifying-the-patched-contracts">Verifying the patched contracts</a></h2>
<p>How do we make sure that we didn’t break a program of perhaps a thousand
lines of code that we’re seeing now for the first time?
It’s especially worrisome, considering that Michelson is not a particularly
human-friendly language. There’s no (formal or even informal)
specification for the contract, no documentation, and all we know about
the author is their public cryptographic key (albeit, in actuality there’s
no guarantee that the person who originated the contract is also the
one who wrote it). The situation seems almost hopeless.</p>
<p>Fortunately one of the particularly terrifying contracts was written
by us some time ago. There was some documentation and even
some tests written in the form of a shell script. So we started with
that one. We rewrote the test, because it relied too much on old
behavior of the <code>tezos-client</code> that has since changed.
Also reworking that test gave us a thorough
understanding of the purpose and interface of the contract. We ran the
test against both the original and the patched version of the contract and
fortunately both versions passed. So far, so good</p>
<p>With other contracts we had less luck though. Fortunately what once
happens on a blockchain, stays there forever. Thus we have a complete
record of transactions ever made to these contracts. We could replay
these transactions with the patched version of the script and check if
they yield the same results as they did originally. How can we replay
a transaction to a smart contract altering some of its aspects? Of
course, we have the <code>tezos-client run script</code> command. It accepts a
script, a storage, and a parameter and executes the given script in the
current context of the node. However, there’s a lot more of the
blockchain’s state to reproduce than just storage and parameter. The
contract has a balance, an address (which is more or less randomly
assigned to the contract during origination). In addition, the transaction comes
from some other account which also has an address and so forth.</p>
<p>Indeed, as we replayed transactions with the original scripts to
verify the soundness of our procedure, we faced some contracts failing
on transactions that succeeded in the past. The apparent point of
failure was checking user signatures. But why would a signature that
used to be valid, become invalid over time? Fortunately, failures
occurred in the contract we had tests for. From these tests we managed
to deduce that the signature consists in encrypting a certain message
with user’s private key, and a part of that message was the address of
the contract being called.</p>
<p>With the <code>run script</code> command the node originated a new contract with a
fresh address to run the script for. Hence the replayed scripts had
a different address than originally and that is why historic signatures
didn’t work in replay. Therefore, it was necessary to force execution with a
given address. So before testing the scripts we had to implement
additional options to the <code>tezos-client</code>. This is how paying off
technical debt can sometimes lead to new features in the software.</p>
<p>Having done that we were able to replay transactions from the past.</p>
<h2 id="scaling-up"><a class="toclink" href="#scaling-up">Scaling up</a></h2>
<p>Of course with all the contracts there were several hundred
transactions to replay in total. Typing the parameters for all these
transactions on the command line takes a lot of time. So it was necessary
to devise an automatic process of replaying the transactions
and checking results.</p>
<p>We decided to use the <span class="caps">API</span> of the <a href="https://tzkt.io">tzkt.io</a> indexer to
download historic transaction data for our contracts. We wrote <a href="https://gitlab.com/sventimir/tezos-replay-transaction">a
script</a> where
for each transaction the local <code>tezos-client</code> is called to execute
the contract. Then, it would parse the transaction result and compare
them to whatever the indexer reported.</p>
<p>Developing the script took a lot of time as we had to translate
between 2 different encodings of Michelson values (<span class="caps">JSON</span> and
Micheline), distinguish between internal operations generated by
the script and the “main” operation that triggered them, and generally
understand the data returned by the indexer in various situations.
We also had to decode operation results returned by the indexer and
compare them to those produced by <code>tezos-client</code>. Lastly, we had to produce
a nice summary, indicating which operations to which contracts failed
so that we could debug them.</p>
<p>After fixing all false-positives and errors we discovered some
operations that we couldn’t get to work. One of the contracts was
previously patched during a migration so that its storage type already
changed. Of course, reproducing transactions from before the patch
was not possible, so we had to skip those<sup id="fnref:storage-type-problem"><a class="footnote-ref" href="#fn:storage-type-problem">3</a></sup>.
Moreover, some contracts relied on big maps, which are stored in
the node’s storage and currently there’s no way to inject their
modified values into the <code>run script</code> command. It’s not at all
obvious how an interface for this should look, not to mention
implementation problems. Hence we decided to give up and leave
these operations unchecked. There were few of them though.</p>
<h2 id="code-review"><a class="toclink" href="#code-review">Code review</a></h2>
<p>The contracts will be patched along with a protocol migration. The node iterates
over all the contracts, looking for hashes of the contracts we want
to modify. Once such a hash is found, the contract for that account is
replaced with a hard-coded binary string representing the patched
version of the contract. How do we review such a change? Binary
representation of Michelson code is even less readable to a human eye
than the code itself. Reviewers could and did review the test for the
one contract that we had a written test for, as well as the migration
code. But how do reviewers read the contract patches if they’re binary-encoded?</p>
<p>To make the review process easier we attached to the
migration code both the original and the patched code for all the modified
contracts as well as diffs between them. We wrote unit tests to verify
that the patched versions were equivalent to hard-coded binary
representations. We also checked that the supplied diffs are the same when
comparing the hard-coded binary representation and the text files
attached to the original code.</p>
<p>Finally we published the <a href="https://gitlab.com/sventimir/tezos-replay-transaction">aforementioned
script</a> to let
the reviewers run the test for themselves. Replaying all the
transactions takes a few hours, so it’s not very convenient to run,
but not that bad if one has to do it only once.</p>
<p>At the time of writing this post, <a href="https://gitlab.com/tezos/tezos/-/merge_requests/3730">the migration
code</a> is
merged into the master branch, and constitutes a part of the <a href="https://research-development.nomadic-labs.com/announcing-jakarta-two.html">Jakarta proposal</a>.
Obviously it won’t be executed until Jakarta is accepted by the
community and activated, though.
That said, having performed all these tests and checks we’re
confident that the patched contracts work the same as they did
before, or at least they do for all the test cases that
reality itself graciously provided us. After all, what better
test to a software than real usage by real users?</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:currently-deprecated">
<p>Actually the word “currently” here refers to
the moment of making the decision, because while working on this, we
already decided to deprecate more features, which were not included in
the scope of this work. <a class="footnote-backref" href="#fnref:currently-deprecated" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:chain-immutability">
<p>Changing the contracts directly would disturb
block hashes, thus making these blocks invalid. <a class="footnote-backref" href="#fnref:chain-immutability" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:storage-type-problem">
<p>The states of the storage that the indexer
remembered were invalid according to the patched version of the
contract, so attempts to replay these transactions with the patched
version resulted in a type error. <a class="footnote-backref" href="#fnref:storage-type-problem" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>Announcing “Jakarta 2”2022-04-27T09:00:00+02:002022-04-27T09:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-04-27:/announcing-jakarta-two.html<!-- spell check: en_US-w_accents -->
<p><em>This is a joint post from Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda, Functori <span class="amp">&</span> Tweag.</em></p>
<p>In a recent <a href="https://research-development.nomadic-labs.com/we-found-two-bugs-in-torus-jakarta.html">blog post</a>, we reported two critical bugs in the <a href="https://research-development.nomadic-labs.com/announcing-tezos-10th-protocol-upgrade-proposal-jakarta.html">original Jakarta protocol proposal</a>, affecting the implementation of the experimental <a href="https://tezos.gitlab.io/protocols/013_jakarta.html#transaction-optimistic-rollups">Transaction Optimistic Rollups <em>(TORUs)</em></a> feature.</p>
<p>Fortunately, we have been able to confidently …</p><!-- spell check: en_US-w_accents -->
<p><em>This is a joint post from Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda, Functori <span class="amp">&</span> Tweag.</em></p>
<p>In a recent <a href="https://research-development.nomadic-labs.com/we-found-two-bugs-in-torus-jakarta.html">blog post</a>, we reported two critical bugs in the <a href="https://research-development.nomadic-labs.com/announcing-tezos-10th-protocol-upgrade-proposal-jakarta.html">original Jakarta protocol proposal</a>, affecting the implementation of the experimental <a href="https://tezos.gitlab.io/protocols/013_jakarta.html#transaction-optimistic-rollups">Transaction Optimistic Rollups <em>(TORUs)</em></a> feature.</p>
<p>Fortunately, we have been able to confidently fix both issues, and a new protocol proposal, <strong>Jakarta 2</strong>, has been since developed, tested and released. Moreover, the latter proposal — whose “real name” is given as usual by its hash, <code>PtJakart2xVj7pYXJBXrqHgd82rdkLey5ZeeGwDgPp9rhQUbSqY</code> — has already <a href="https://tzstats.com/election/head">been injected</a> and it is already competing with the original Jakarta one in the current Proposal period.</p>
<p>Yesterday, Nomadic Labs published <a href="https://forum.tezosagora.org/t/release-candidate-of-octez-13-0-rc1/4509">a release candidate</a> for a new Octez suite version, <code>v13.0~rc1</code>. This release candidate includes the daemon binaries that enable testing the <strong>Jakarta 2</strong> protocol proposal in the <a href="https://teztnets.xyz/jakartanet-about">upcoming <code>Jakartanet</code> test network</a>.</p>
<h2 id="changes-in-jakarta-2"><a class="toclink" href="#changes-in-jakarta-2">Changes in Jakarta 2</a></h2>
<p>The changes in <strong>Jakarta 2</strong> with regard to the previous Jakarta proposal concern only fixing the reported bugs. In this section we shed more light into these issues, and detail how they were corrected for the new protocol proposal.</p>
<h3 id="bugfix-for-uncraftable-rejection-operations"><a class="toclink" href="#bugfix-for-uncraftable-rejection-operations">Bugfix for uncraftable rejection operations</a></h3>
<p>The first bug concerns <em>rejection operations</em>, that is the Layer 1 operation which allows a honest node to refute an erroneous or malicious commitment in a <span class="caps">TORU</span>. A flaw in its implementation ultimately made it possible for an attacker to publish commitments whose refutation proofs would not fit within the maximal allowed size for Tezos’ <em>manager</em><sup id="fnref:manager"><a class="footnote-ref" href="#fn:manager">1</a></sup> operations batches — 32 KiB.</p>
<p>This limit underpins a design decision in the current implementation of TORUs: there is an upper-bound on the number of Layer 2 transaction batches from the same rollup which can be (ahem!) rolled up within a given Tezos Layer 1 block. For each of these batches, operators of the client rollup are required to include the batch’s hash in their respective commitment operation on Layer 1. Inserting too many batches from the same rollup in the same block would make it impossible to insert a valid commitment for it in a subsequent block, which would result in the incriminated rollup being stuck forever.</p>
<p>Rejection operations denounce a specific rollup commitment, which contains one hash per batch of Layer 2 transactions. Honest rollup accusers can trivially find the first hash that they disagree with, and craft a rejection operation to refute said batch. The rejection operations requires two key pieces of data: (1) the target batch itself, and (2) a collection of Merkle proofs which allow to replay said message. This raises an immediate concern. What if the provided Merkle proofs <em>are just too large</em> to fit within the size boundaries a Tezos Layer 1 manager operation? If the batch is large enough (that is, it contains enough transactions), this can definitely happen.</p>
<p>To tackle this risk, TORUs relies on two design choices. First, <strong>TORUs encode Merkle proofs using a format that allows to send <em>truncated</em> proofs</strong>. Second, <strong>a batch for which one can exhibit a valid, “large enough” truncated proof has to be considered a no-op</strong>.</p>
<p>In the first Jakarta proposal, the size boundary for Layer 2 operations batches was 5 <span class="caps">KB</span>, and a <em>truncated proof</em> larger than 30 <span class="caps">KB</span> was required, in order to prove that a batch should be considered a <code>no-op</code>. So, in order for an honest rollup node to prove that a batch of size <span class="caps">5KB</span> was to be ignored, it needed to produce a refutation whose size was at least 35 <span class="caps">KB</span> (~36 KiB)… which does not fit within the 32 KiB limit on Layer 1.</p>
<p>The fix is straightforward: accept a truncated proof of <code>30KiB - sizeof(batch)</code>. In this way, rejection operations are guaranteed to fit within a single Tezos manager operation. The curious reader can have a look at the <a href="https://gitlab.com/tezos/tezos/-/merge_requests/5080/diffs?commit_id=ad065359ee57cd3df5673e7e7de421e63942985b">diff of the <span class="caps">MR</span> implementing this bugfix</a>.</p>
<h3 id="bugfix-for-forged-zero-valued-tickets"><a class="toclink" href="#bugfix-for-forged-zero-valued-tickets">Bugfix for forged zero-valued Tickets</a></h3>
<p>The second bug concerns Tezos <a href="https://tezos.gitlab.io/jakarta/michelson.html#operations-on-tickets">Tickets</a>. The latter are an abstract representation of tokenized assets, and in TORUs, they are used to represent the assets deposited and transacted within the rollup.</p>
<p>Tickets can be seen as a tuple consisting of: (1) its (typed) payload, (2) the address of the smart contract which has minted it — a.k.a its <em>ticketer</em> — , and (3) a quantity. A ticket can be split into two (and eventually more parts), assuming that the total sum of the quantities associated to each of the two resulting half-tickets is equal to the amount of the original one. Dually, two tickets with the same payload and issued by the same ticketer, can also be joined, adding up the quantities.</p>
<p>A subtle, and often-overlooked detail is that a zero-valued Ticket is a completely valid one. <strong>In a nutshell, owning a ticket whose quantity is 0 is not the same thing as not owning any ticket at all — and the consequences or the semantic meaning of the former depends on the application.</strong></p>
<p>Both with TORUs and the Tezos protocol’s <em>Table of Tickets</em><sup id="fnref:table"><a class="footnote-ref" href="#fn:table">2</a></sup> specific design choices concerning zero-valued tickets were made, in order to simplify each component’s designs. On one hand, the <em>Table of Tickets</em> does not take into account zero-valued tickets at all. This is a legitimate choice, because its purpose is to prevent the <em>forgery</em>, and the incorrect <em>splitting</em>, of tickets holding non-zero quantities. On the other hand, in TORUs we have decided to <strong>not support zero-valued Ticket transfers within the Layer 2</strong>, and to rely on the Table of Tickets to transfer ticket ownership in the Layer 1. In the end, these choices were proven to be incompatible, was the <span class="caps">TORU</span> implementation relied on stronger assumptions than those provided by the Table of Tickets. As it is often the case in Software Engineering, composing two software components brought unforeseen issues in the resulting system.</p>
<p>In Jakarta 2, we addressed this situation by enforcing in a stricter manner that zero-valued tickets are not supported within the rollups, adding additional checks in the Tezos protocol’ Layer 1. More precisely, <a href="https://gitlab.com/tezos/tezos/-/merge_requests/5078/diffs?commit_id=f89bfe1c6470bfc0c246f5bec163951c0477abd9#diff-content-dd08bf62aa0d2024bb90b1fe653c7298bb8c9556">three straightforward bound checks are enough to prevent zero-valued tickets being forged by TORUs withdrawal operations</a>.</p>
<h2 id="join-jakartanet"><a class="toclink" href="#join-jakartanet">Join Jakartanet!</a></h2>
<p>A testnet for the Jakarta 2 protocol proposal, named <code>Jakartanet</code>, <a href="https://teztnets.xyz/jakartanet-about">will launch on April 27th at 15hs <span class="caps">UTC</span></a>. It is critical to have as many bakers as possible participating in this testnet, by running nodes, producing blocks and
deploying apps. We are looking for more bakers to participate from day one. If you are interested, please <a href="https://tezos-kiln.org/joinbakingslack">join the Tezos Baking slack</a> and make yourselves known in the <code>#test-networks</code> channel.</p>
<p>Once more, we strongly encourage you to test your own Tezos-based applications to check for compatibility problems.</p>
<p>Should the Jakarta 2 protocol proposal be accepted by the community, the following minimal version of Tezos node (<em>shell</em>) software will be necessary to participate in the consensus, due to necessary changes introduced to the protocol environment: <code>v13</code> of Octez, or <code>v3</code> of TezEdge.</p>
<!-- THOMAS ORIGINAL EXCUSES
### Uncraftable Rejection Operation
You may or may not be aware of this, but Tezos does enforce a strict upper-bound on the size of the operations that can appear in a block. In Jakarta, this upper-bound has not changed, and remain set to 32KiB.
One of the main constrains of the Transaction Rollups implementation is to ensure that the arguments of the layer-1 operations introduced in Jakarta **always** fit within the boundaries of a valid Tezos (manager) operation. This is why there is an upper-bound on the number of layer-2 batches that can be submitted in a given Tezos block: because for each batch, you need to add a hash in the respective commitment. Inserting too many batches in a Tezos block would make it impossible to insert a valid commitment in a subsequent Tezos block, which would block the incriminated rollup forever.
Similar concerns apply to the rejection operation, albeit for different reasons. First, a quick crash course on how the rejection operation works. Because a commitment contains one hash per batch, accusers can trivially find the first hash that they disagree with. For this batch, they are expected to provide two key data: (1) the batch itself, and (2) a collection of Merkle proofs which allow to replay said message. This raises an immediate concern. What if the Merkle proofs _are just too large_ to fit in a Tezos operation? If the batch is large enough (that is, it contains enough transactions), this can definitely happen. To tackle this risk, TORU relies on two design choices. Firstly, **TORU encodes Merkle proofs with a format that allows to send truncated proofs**. Secondly, **a batch for which one can exhibit a valid, truncated proofs “large enough” has to be considered a no-op**.
In Jakarta, a layer-2 batch cannot be larger than 5KB, and a truncated proof larger than 30KB is required to prove that a batch should be considered a no-op… so to prove that a 5KB batch has to be ignored, one needs at least 35KB… that cannot fit in 32KiB.
The fix is straightforward: accept a truncated proof of `30KB - sizeof(batch)`. This way, we are guaranteed to be able to fit everything in a Tezos operation. The curious reader can have a look at the [diff of the MR containing the fix](https://gitlab.com/tezos/tezos/-/merge_requests/5080/diffs?commit_id=ad065359ee57cd3df5673e7e7de421e63942985b).
## Tickets Forgery
In Tezos, a ticket is a triplet of (1) a (typed) contents, (2) the address of the smart contract which has minted the ticket –it’s _ticketer_–, and (3) a quantity. A ticket can be split into two, assuming the sum of the quantity of the two resulting tickets is equal to the quantity of the original one. Besides, two tickets with the same contents and ticketer can also be joined (the dual operation of splitting).
An aspect of Tezos ticket that is too often overlooked is that tickets with a quantity equal to zero are totally valid. In a nutshell, owning a ticket of quantity 0 (a _0-ticket_) is not the same thing as not owning any ticket at all.
Both TORU and the table of tickets have made specific design choices concerning 0-tickets, in order to simplify their respective designs. On the one hand, the table of tickets does not take into account 0-tickets at all. This is a legitimate choice, because the main flaw the table of tickets is expected to counter is ticket duplication, not ticket forgery. On the other hand, TORU has decided to not support 0-ticket transfers in the layer-2, and to rely on the table of tickets to transfer tickets’ ownership in the layer-1. In the end, these choices have proven to create a bad mix, wherein the prerequisites the TORU implementation had on the table of tickets weren’t met. As often in software engineering, composing two software components brought unforeseen issues in the resulting system.
For this one, the fix is to enforce in a stricter manner that 0-tickets are not supported, with additional checks in the layer-1. More precisely, [three straightforward bound check were enough to prevent TORU’s operations to forge tickets](https://gitlab.com/tezos/tezos/-/merge_requests/5078/diffs?commit_id=f89bfe1c6470bfc0c246f5bec163951c0477abd9#diff-content-dd08bf62aa0d2024bb90b1fe653c7298bb8c9556).
--->
<div class="footnote">
<hr>
<ol>
<li id="fn:manager">
<p>Manager operations is the term we use in the Tezos architecture to denote any <em>fee-paying</em> operation in competition for block space, like transfers, smart contract originations, and smart contracts calls. Thus, the new Layer 1 operations related to rollups belong to this category as well (see <a href="https://tezos.gitlab.io/jakarta/protocol_overview.html#operations-validation-passes">here</a> for more details). <a class="footnote-backref" href="#fnref:manager" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:table">
<p>the Table of Tickets, introduced in Jakarta, is an internal component of the Tezos economic protocol who keeps track of all emitted tickets in the current context — that is at the current head of the blockchain. <a class="footnote-backref" href="#fnref:table" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>We discovered two bugs in Jakarta — a reproposal is coming2022-04-22T17:00:00+02:002022-04-22T17:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-04-22:/we-found-two-bugs-in-torus-jakarta.html<p>During continued stress-testing of the Jakarta protocol proposal, we discovered two critical bugs in the implementation of Transaction Optimistic Rollups. A new proposal —Jakarta 2— will be released in time to be considered in the ongoing proposal period. We encourage bakers to avoid upvoting of the initial Jakarta proposal, and to upvote Jakarta 2 instead.</p><!-- spell check: en_US-w_accents -->
<p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: During continued stress-testing of the Jakarta protocol
proposal, we discovered two critical bugs in the implementation of
Transaction Optimistic Rollups. A new proposal —Jakarta 2— will be
released in time to be considered in the ongoing proposal period. We
encourage bakers to avoid upvoting of the initial Jakarta proposal,
and to upvote Jakarta 2 instead.</strong></p>
<p><strong>Update: Jakarta 2 has been released. More details in <a href="https://research-development.nomadic-labs.com/announcing-jakarta-two.html">its
announcement blog
post</a>.</strong></p>
<p>We recently <a href="https://research-development.nomadic-labs.com/announcing-tezos-10th-protocol-upgrade-proposal-jakarta.html">announced
Jakarta</a>,
a new protocol proposal for the Tezos blockchain which notably
introduces the first, experimental, step in <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html">our scalability
roadmap</a>:
<a href="https://tezos.gitlab.io/jakarta/transaction_rollups.html">transaction optimistic rollups
(TORUs)</a>.</p>
<p>Despite significant unit testing, property testing, and integration
testing of the new protocol features prior to injection, we have found
<strong>two critical bugs</strong> in the implementation of TORUs during continued
post-injection testing.</p>
<p>The first bug is an issue with the <em>rejection operation</em>, that is the
Layer 1 operation which allows a honest node to refute an erroneous
(read, malicious) commitment. A flaw in its implementation ultimately
makes it possible for an attacker to publish commitments for which
honest participants cannot provide refutation proofs. <strong>This is a
critical security bug, but one isolated to TORUs. It has no impact on
the security of Tezos’ Layer 1</strong>.</p>
<p>The second bug concerns <a href="https://tezos.gitlab.io/jakarta/michelson.html#operations-on-tickets">Ticket
withdrawals</a>,
the underlying mechanism allowing for tokens to be transferred back
from the Layer 2 rollup to Layer 1. The bug enables an attacker to
forge tickets accounting for 0 tokens.</p>
<!— Tickets being an abstract representation of tokenized assets, the
semantics of zero-valued tickets depend on each of the smart contracts
that legitimately minted them. However, this bug renders any ticket
accounting for 0 tokens inherently suspicious, and ultimately
worthless. **Thus, this is a critical bug which might affect existing
or future applications building on Tezos’ Tickets**. ->
<!— ANDI: I don’t understand this paragraph. Not sure if just lack of
technical knowledge. What is a zero-valued ticket and why can they be
forged? GAD: A zero-valued ticket is a ticket amounting for 0 tokens
— in some use cases it does (or may) makes sense, the semantics of
tickets allows it. They can be forged as a result of the bug in
Jakarta. So if you have a smart contract where zero-valued tickets
mean something, the bug allows an attacker to exploit it to create
them for free. So, nasty. —>
<p>In TORUs, tickets are used to internally represent assets deposited to
the rollup. The design of tickets allows for zero-valued ticket
accounting for 0 tokens, which in some use cases can make
sense. However, due to the bug in Jakarta, zero-value tickets can be
forged when withdrawing from a rollup. For other smart contracts where
zero-valued tickets are indeed meaningful, the bug can be exploited by
an attacker to create these for free. Hence, the bug renders any
ticket accounting for 0 tokens inherently suspicious, and ultimately
worthless. <strong>Thus, this is a critical bug which might affect existing
or future applications building on Tezos’ Tickets</strong>.</p>
<!-- GAD -> Andy: tickets can exist outside rollups, actually the
attack is on Layer 1. Tickets are used in TORUs to implement the
native token internal to the rollup. ANDI -> GAD + others: Not sure
if the above (commented out) take makes sense, just delete if not GAD:
I made two small nits, pinging Thomas.
-->
<p><strong>Fortunately, we have been able to confidently fix both issues, using
fairly small patches that were easy to review, and that are backed up
with appropriate tests, which have now been added to the existing test suite.</strong></p>
<h2 id="a-call-to-bakers"><a class="toclink" href="#a-call-to-bakers">A call to bakers</a></h2>
<p>While the original Jakarta protocol proposal has already passed the
minimal quorum, we are still early in the <a href="https://www.tezosagora.org/period/70">proposal
period</a>. There is sufficient
time for Jakarta 2 to achieve enough upvotes for it to replace the
original proposal as the one that will continue to the next step.</p>
<p><em>We encourage bakers who have still not voted to wait until the new
Jakarta 2 proposal is released early next week. We also encourage
bakers who have already voted for Jakarta to vote again for the new
proposal once <!--and if--> it is injected.</em></p>
<p>We have decided to delay the publication of the release candidate for
the next Octez version <code>v13~rc1</code>, so as to provide the baker and
accuser binaries for Jakarta 2 instead. The
<a href="https://teztnets.xyz/jakartanet-about"><code>Jakartanet</code></a> test network
will thus be based on the <code>Jakarta 2</code> protocol proposal.</p>
<p>We invite the community, specially developers of applications building
upon TORUs, to join <code>Jakartanet</code> from the beginning. Moreover, we want
to take the opportunity to advertise and advocate the usage of the</p>
<!-- rolling, bleeding edge, -->
<p><a href="https://teztnets.xyz/">Dailynet and Mondaynet
teztnets</a> — feel free to reach out to us if
you need help deploying your contracts and applications on the test networks.</p>
<p>As for future proposals, we have already taken actions to improve the
specification and testing frameworks to better uncover similar flaws,
and we intend to strengthen them even more. The occasional bug is
unavoidable, some will be critical and others less so, but we strive
to make it continuously less likely that larger bugs escape the
nets. And when they do, its unique on-chain amendment process makes
the Tezos blockchain uniquely equipped to react, fix, and upgrade accordingly.</p>
<!--ANDI: Would not say on-chain gov makes it easier to fix GAD: my
point was to say we can fix and patch before them hitting mainnet, and
thus avoid forks. I Put it back in, let's see how folks react.-->TPS evaluation for Tezos2022-04-19T18:00:00+02:002022-04-19T18:00:00+02:00Mark Karpov (Tweag) and Nomadic Labstag:research-development.nomadic-labs.com,2022-04-19:/tps-evaluation.html<p>An analysis of Tezos throughput</p><!--- en-US spellcheck -->
<p>In this post we are going to try to answer the question “what is the
throughput of Tezos on the Mainnet” by examining the subject from different
angles. First, we must note that there is no standard methodology for
measuring the throughput of a blockchain. In general, people want to know
the number of <em>transactions per second</em> (<span class="caps">TPS</span>) that a blockchain can process.
There are, however, a few questions that need to be answered first:</p>
<ul>
<li>What is the configuration of the network that is being examined? Is it a
minimal network or a larger network where latencies are increasingly important?</li>
<li>What does it mean for a transaction to be “processed”? Does it mean that
it is included in the next produced block or perhaps in a decided block?</li>
<li>Is the <span class="caps">TPS</span> value limited by the protocol constants/parameters or technical
limitations of the system?</li>
<li>What kind of transactions are we dealing with? In most systems that
support smart contracts the makeup of the stream of transactions that is
being processed is going to directly affect the maximal <span class="caps">TPS</span> number. The
situation is simpler with e.g. Bitcoin where all transactions are
identical from that point of view.</li>
</ul>
<p>The last point is an important one because, all things being equal, <span class="caps">TPS</span> of a
blockchain can change over time depending on the transactions users tend to
perform. Let’s take a closer look.</p>
<h2 id="analyzing-transactions"><a class="toclink" href="#analyzing-transactions">Analyzing transactions</a></h2>
<p>In Tezos, all transactions can be divided into two groups depending on their destination:</p>
<ol>
<li>Transactions that have an implicit contract as destination.</li>
<li>Transactions that have a smart contract as destination.</li>
</ol>
<p>In the first case the cost (both in gas and in terms of processing power) is
constant no matter the amount being transferred and other parameters. In the
second case, however, the cost and the performance of such a transaction is
going to depend on the code of the smart contract in question.</p>
<p>It follows, that in order to determine the <span class="caps">TPS</span>, one needs to know the
proportions of different kinds of transactions in the typical
block. The best way to achieve that is to analyze the history of
performed transactions. We choose to use the <a href="https://gitlab.com/nomadic-labs/tezos-indexer">Tezos
indexer</a> to do
that. The tool stores Tezos’s transaction history in a Postgresql
database, which we can query and analyze. The result is a <span class="caps">JSON</span> file
such as this one:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span><span class="m">2022</span>-01-01-to-2022-02-28.json
<span class="o">{</span>
<span class="w"> </span><span class="s2">"regular"</span>:<span class="w"> </span><span class="m">5026035</span>,
<span class="w"> </span><span class="s2">"origination"</span>:<span class="w"> </span><span class="m">16905</span>,
<span class="w"> </span><span class="s2">"contract"</span>:<span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="s2">"KT1RJ6PbjHpwc3M5rw5s2Nbmefwbuwbdxton"</span>:<span class="w"> </span><span class="m">1611069</span>
<span class="w"> </span><span class="o">}</span>
<span class="o">}</span>
</code></pre></div>
<p>Here, we can see that in the first two months of 2022 there were 5026035
transactions that had an implicit contract as their destination and the most
popular smart contract at the time was
<code>KT1RJ6PbjHpwc3M5rw5s2Nbmefwbuwbdxton</code> (Hic et Nunc NFTs) with 1611069
calls. For now, we include only smart contracts that are involved in at
least 10% of all transactions, but this threshold can be changed. Having
this data, we can run gas estimations for regular transfers and smart
contracts calls in order to calculate the average transaction cost.</p>
<h2 id="gas-based-estimation"><a class="toclink" href="#gas-based-estimation">Gas-based estimation</a></h2>
<p>It is possible to perform a <span class="caps">TPS</span> estimation based on the maximal
allowed gas consumption per block. Transactions in a block cannot
exceed <code>hard_gas_limit_per_block</code>, which is a protocol parameter; its
value (on the <code>master</code> branch) is 5200000. With
<a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">Tenderbake</a>,
in the best case scenario a block will be decided at round 0,
therefore, according to the current value of <code>minimal_block_delay</code>,
the maximal block production rate is 1 block per 30 seconds. Next, we
can use the average transaction cost that we have obtained through
analysis of transaction data to determine <span class="caps">TPS</span> as:</p>
<div class="highlight"><pre><span></span><code>tps = hard_gas_limit_per_block / (average_transaction_cost * minimal_block_delay)
</code></pre></div>
<p>Our utility (which is part of Tezos and can be built by running <code>make
build-tps-deps && make build-tps</code>) can perform this estimation:</p>
<div class="highlight"><pre><span></span><code><span class="p">.</span><span class="o">/</span><span class="n">tezos</span><span class="o">-</span><span class="n">tps</span><span class="o">-</span><span class="n">evaluation</span><span class="o">-</span><span class="n">gas</span><span class="o">-</span><span class="n">tps</span><span class="w"> </span><span class="o">-</span><span class="n">a</span><span class="w"> </span><span class="n">average</span><span class="o">-</span><span class="n">block</span><span class="o">=</span><span class="n">src</span><span class="o">/</span><span class="n">bin_tps_evaluation</span><span class="o">/</span><span class="n">average</span><span class="o">-</span><span class="n">block</span><span class="p">.</span><span class="n">json</span>
<span class="o">[</span><span class="n">14:26:25.243</span><span class="o">]</span><span class="w"> </span><span class="n">Starting</span><span class="w"> </span><span class="nl">test</span><span class="p">:</span><span class="w"> </span><span class="n">tezos_tps_gas</span>
<span class="o">[</span><span class="n">14:26:27.956</span><span class="o">]</span><span class="w"> </span><span class="n">Reading</span><span class="w"> </span><span class="n">description</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">average</span><span class="w"> </span><span class="n">block</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">src</span><span class="o">/</span><span class="n">bin_tps_evaluation</span><span class="o">/</span><span class="n">average</span><span class="o">-</span><span class="n">block</span><span class="p">.</span><span class="n">json</span>
<span class="o">[</span><span class="n">14:26:28.061</span><span class="o">]</span><span class="w"> </span><span class="n">Originating</span><span class="w"> </span><span class="n">smart</span><span class="w"> </span><span class="n">contracts</span>
<span class="o">[</span><span class="n">14:26:28.285</span><span class="o">]</span><span class="w"> </span><span class="n">Waiting</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">reach</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="k">next</span><span class="w"> </span><span class="k">level</span>
<span class="o">[</span><span class="n">14:26:57.514</span><span class="o">]</span><span class="w"> </span><span class="n">Average</span><span class="w"> </span><span class="k">transaction</span><span class="w"> </span><span class="nl">cost</span><span class="p">:</span><span class="w"> </span><span class="mi">2900</span>
<span class="o">[</span><span class="n">14:26:57.514</span><span class="o">]</span><span class="w"> </span><span class="n">Gas</span><span class="w"> </span><span class="nl">TPS</span><span class="p">:</span><span class="w"> </span><span class="mi">60</span>
<span class="o">[</span><span class="n">14:26:57.536</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">SUCCESS</span><span class="o">]</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="o">/</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="n">tezos_tps_gas</span>
</code></pre></div>
<p>We can see that the gas-based estimation of <span class="caps">TPS</span> on Mainnet is currently 60.</p>
<p>For comparison with blockchains like Bitcoin that do not support smart
contracts, here is a gas-based estimation for regular transactions only:</p>
<div class="highlight"><pre><span></span><code><span class="p">.</span><span class="o">/</span><span class="n">tezos</span><span class="o">-</span><span class="n">tps</span><span class="o">-</span><span class="n">evaluation</span><span class="o">-</span><span class="n">gas</span><span class="o">-</span><span class="n">tps</span>
<span class="o">[</span><span class="n">08:40:21.806</span><span class="o">]</span><span class="w"> </span><span class="n">Starting</span><span class="w"> </span><span class="nl">test</span><span class="p">:</span><span class="w"> </span><span class="n">tezos_gas_tps</span>
<span class="o">[</span><span class="n">08:40:21.807</span><span class="o">]</span><span class="w"> </span><span class="n">Gas</span><span class="w"> </span><span class="n">TPS</span><span class="w"> </span><span class="n">estimation</span>
<span class="o">[</span><span class="n">08:40:24.382</span><span class="o">]</span><span class="w"> </span><span class="k">Using</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="k">default</span><span class="w"> </span><span class="n">average</span><span class="w"> </span><span class="n">block</span><span class="w"> </span><span class="n">description</span>
<span class="o">[</span><span class="n">08:40:24.499</span><span class="o">]</span><span class="w"> </span><span class="n">Originating</span><span class="w"> </span><span class="n">smart</span><span class="w"> </span><span class="n">contracts</span>
<span class="o">[</span><span class="n">08:40:24.723</span><span class="o">]</span><span class="w"> </span><span class="n">Waiting</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">reach</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="k">next</span><span class="w"> </span><span class="k">level</span>
<span class="o">[</span><span class="n">08:40:54.642</span><span class="o">]</span><span class="w"> </span><span class="n">Average</span><span class="w"> </span><span class="k">transaction</span><span class="w"> </span><span class="nl">cost</span><span class="p">:</span><span class="w"> </span><span class="mi">1421</span>
<span class="o">[</span><span class="n">08:40:54.643</span><span class="o">]</span><span class="w"> </span><span class="n">Gas</span><span class="w"> </span><span class="nl">TPS</span><span class="p">:</span><span class="w"> </span><span class="mi">122</span>
<span class="o">[</span><span class="n">08:40:54.667</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">SUCCESS</span><span class="o">]</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="o">/</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="n">tezos_gas_tps</span>
</code></pre></div>
<h2 id="benchmark-based-estimation"><a class="toclink" href="#benchmark-based-estimation">Benchmark-based estimation</a></h2>
<p>It is also possible to lift limits imposed by the protocol in the form of
constants and parameters, as well as hard-coded values in order to determine
the purely technical <span class="caps">TPS</span> limit by running a specialized benchmark. This
approach requires us to make some choices regarding the setup:</p>
<ul>
<li>We are going to examine the smallest self-sufficient network possible: a
node + a baker. This way we will be able to more directly judge raw
performance of the node and the baker, but not of a distributed network.
The former is much simpler and more predictable than the latter. Also,
needless to say, performance of a distributed network depends directly on
the performance of the node.</li>
<li>A transaction is considered processed if it is successfully injected and
included in the next produced block, i.e. it has been successfully applied
and validated.</li>
<li>The stream of transactions is modeled on actual historical data.</li>
</ul>
<p>The <span class="caps">TPS</span> benchmark is run with the <code>tezos-tps-evaluation-benchmark-tps</code>
command. It spawns a network comprising a node, a baker, and a client. The
network will use the same constants and parameters as the Mainnet. By
default 10 blocks will be produced, but this can be changed by supplying the
<code>blocks-total</code> command line option. The total number of applied operations
in these blocks will be divided by the total time spent producing the blocks
and the resulting value will be presented as the <strong>empirical <span class="caps">TPS</span></strong>. The
benchmark is also capable of calculating <strong>de facto <span class="caps">TPS</span> of injection</strong>—the
number of transactions actually injected, which is useful in judging the results.</p>
<div class="highlight"><pre><span></span><code><span class="p">.</span><span class="o">/</span><span class="n">tezos</span><span class="o">-</span><span class="n">tps</span><span class="o">-</span><span class="n">evaluation</span><span class="o">-</span><span class="n">benchmark</span><span class="o">-</span><span class="n">tps</span><span class="w"> </span><span class="o">-</span><span class="n">a</span><span class="w"> </span><span class="n">average</span><span class="o">-</span><span class="n">block</span><span class="o">=</span><span class="n">src</span><span class="o">/</span><span class="n">bin_tps_evaluation</span><span class="o">/</span><span class="n">average</span><span class="o">-</span><span class="n">block</span><span class="p">.</span><span class="n">json</span><span class="w"> </span><span class="o">-</span><span class="n">a</span><span class="w"> </span><span class="n">lift</span><span class="o">-</span><span class="n">protocol</span><span class="o">-</span><span class="n">limits</span>
<span class="o">[</span><span class="n">20:42:19.167</span><span class="o">]</span><span class="w"> </span><span class="n">Starting</span><span class="w"> </span><span class="nl">test</span><span class="p">:</span><span class="w"> </span><span class="n">tezos_tps_benchmark</span>
<span class="o">[</span><span class="n">20:42:19.167</span><span class="o">]</span><span class="w"> </span><span class="n">Gas</span><span class="w"> </span><span class="n">TPS</span><span class="w"> </span><span class="n">estimation</span>
<span class="o">[</span><span class="n">20:42:21.761</span><span class="o">]</span><span class="w"> </span><span class="n">Reading</span><span class="w"> </span><span class="n">description</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">average</span><span class="w"> </span><span class="n">block</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">src</span><span class="o">/</span><span class="n">bin_tps_evaluation</span><span class="o">/</span><span class="n">average</span><span class="o">-</span><span class="n">block</span><span class="p">.</span><span class="n">json</span>
<span class="o">[</span><span class="n">20:42:21.858</span><span class="o">]</span><span class="w"> </span><span class="n">Originating</span><span class="w"> </span><span class="n">smart</span><span class="w"> </span><span class="n">contracts</span>
<span class="o">[</span><span class="n">20:42:22.082</span><span class="o">]</span><span class="w"> </span><span class="n">Waiting</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">reach</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="k">next</span><span class="w"> </span><span class="k">level</span>
<span class="o">[</span><span class="n">20:42:51.262</span><span class="o">]</span><span class="w"> </span><span class="n">Average</span><span class="w"> </span><span class="k">transaction</span><span class="w"> </span><span class="nl">cost</span><span class="p">:</span><span class="w"> </span><span class="mi">2900</span>
<span class="o">[</span><span class="n">20:42:51.262</span><span class="o">]</span><span class="w"> </span><span class="n">Gas</span><span class="w"> </span><span class="nl">TPS</span><span class="p">:</span><span class="w"> </span><span class="mi">60</span>
<span class="o">[</span><span class="n">20:42:51.293</span><span class="o">]</span><span class="w"> </span><span class="n">Tezos</span><span class="w"> </span><span class="n">TPS</span><span class="w"> </span><span class="n">benchmark</span>
<span class="o">[</span><span class="n">20:42:51.293</span><span class="o">]</span><span class="w"> </span><span class="nl">Protocol</span><span class="p">:</span><span class="w"> </span><span class="n">Alpha</span>
<span class="o">[</span><span class="n">20:42:51.293</span><span class="o">]</span><span class="w"> </span><span class="n">Blocks</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="nl">bake</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span>
<span class="o">[</span><span class="n">20:42:51.293</span><span class="o">]</span><span class="w"> </span><span class="n">Accounts</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="k">use</span><span class="err">:</span><span class="w"> </span><span class="mi">30000</span>
<span class="o">[</span><span class="n">20:42:51.293</span><span class="o">]</span><span class="w"> </span><span class="n">Spinning</span><span class="w"> </span><span class="n">up</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">network</span><span class="p">...</span>
<span class="o">[</span><span class="n">20:43:01.039</span><span class="o">]</span><span class="w"> </span><span class="n">Originating</span><span class="w"> </span><span class="n">smart</span><span class="w"> </span><span class="n">contracts</span>
<span class="o">[</span><span class="n">20:43:01.861</span><span class="o">]</span><span class="w"> </span><span class="n">Waiting</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">reach</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="k">next</span><span class="w"> </span><span class="k">level</span>
<span class="o">[</span><span class="n">20:43:27.301</span><span class="o">]</span><span class="w"> </span><span class="k">Using</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="k">parameter</span><span class="w"> </span><span class="k">file</span><span class="err">:</span><span class="w"> </span><span class="o">/</span><span class="n">run</span><span class="o">/</span><span class="k">user</span><span class="o">/</span><span class="mi">1000</span><span class="o">/</span><span class="n">tezt</span><span class="o">-</span><span class="mi">185969</span><span class="o">/</span><span class="mi">1</span><span class="o">/</span><span class="k">parameters</span><span class="p">.</span><span class="n">json</span>
<span class="o">[</span><span class="n">20:43:27.301</span><span class="o">]</span><span class="w"> </span><span class="n">Waiting</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">reach</span><span class="w"> </span><span class="k">level</span><span class="w"> </span><span class="mi">3</span>
<span class="o">[</span><span class="n">20:43:57.054</span><span class="o">]</span><span class="w"> </span><span class="n">The</span><span class="w"> </span><span class="n">benchmark</span><span class="w"> </span><span class="n">has</span><span class="w"> </span><span class="n">been</span><span class="w"> </span><span class="n">started</span>
<span class="o">[</span><span class="n">20:49:03.482</span><span class="o">]</span><span class="w"> </span><span class="n">Produced</span><span class="w"> </span><span class="mi">10</span><span class="w"> </span><span class="n">block</span><span class="p">(</span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="mf">306.43</span><span class="w"> </span><span class="n">seconds</span>
<span class="o">[</span><span class="n">20:49:04.091</span><span class="o">]</span><span class="w"> </span><span class="n">BLSh3JB76aHp2Zg37zrv35kr6XNu8Nti9hiGELriEWNJXHoEQN9</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="mi">5699</span>
<span class="o">[</span><span class="n">20:49:04.431</span><span class="o">]</span><span class="w"> </span><span class="n">BM2MMEawEUvKZm25LoB81rGEm2MtQtdkaXzAeuJACsbcZcsuuD2</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="mi">5651</span>
<span class="o">[</span><span class="n">20:49:04.759</span><span class="o">]</span><span class="w"> </span><span class="n">BLRN2oxtgpep1D3oCiD6NGo9GPha6Z2S69LiTa8SPskHLdjN1Ag</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="mi">5762</span>
<span class="o">[</span><span class="n">20:49:05.095</span><span class="o">]</span><span class="w"> </span><span class="n">BMD3U3mv8EM9ZKYsK6sxFob5ZHHCYRUZiSn6LJVgvnp8eMaV8bY</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="mi">5822</span>
<span class="o">[</span><span class="n">20:49:05.411</span><span class="o">]</span><span class="w"> </span><span class="n">BLCZf4mHGaawCCuJZLF2yFXwhg1tn32BbBezYm4dnik1mfDQ4wK</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="mi">5892</span>
<span class="o">[</span><span class="n">20:49:05.684</span><span class="o">]</span><span class="w"> </span><span class="n">BLpPYyppLmp4enYopGBWNHvhGLzBFuWaFSJmEqkR1QQK1stWbPt</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="mi">3868</span>
<span class="o">[</span><span class="n">20:49:05.996</span><span class="o">]</span><span class="w"> </span><span class="n">BKmqoWrZgevEmmd7RXaEX5umsBQaW3cnHJRK4AJq5uzEXzaryeE</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="mi">5721</span>
<span class="o">[</span><span class="n">20:49:06.303</span><span class="o">]</span><span class="w"> </span><span class="n">BMGpJV4HtYjzfeq5HLJmvN9LFpPoPeuBeEdGdwYsZ57rc7gyvnQ</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="mi">5810</span>
<span class="o">[</span><span class="n">20:49:06.610</span><span class="o">]</span><span class="w"> </span><span class="n">BLYCC8jRAb4xAQAYzZixC4PSYLacFtFHAV2C6jWi43obZL9aXVw</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="mi">5537</span>
<span class="o">[</span><span class="n">20:49:06.918</span><span class="o">]</span><span class="w"> </span><span class="n">BLAb72vxpN5tnGoq2gbLP1ce8XfAqEruWgbkheyKdiUmMC5LuDt</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="mi">5417</span>
<span class="o">[</span><span class="n">20:49:06.918</span><span class="o">]</span><span class="w"> </span><span class="n">Total</span><span class="w"> </span><span class="n">applied</span><span class="w"> </span><span class="nl">transactions</span><span class="p">:</span><span class="w"> </span><span class="mi">55179</span>
<span class="o">[</span><span class="n">20:49:06.918</span><span class="o">]</span><span class="w"> </span><span class="n">Total</span><span class="w"> </span><span class="n">injected</span><span class="w"> </span><span class="nl">transactions</span><span class="p">:</span><span class="w"> </span><span class="mi">55452</span>
<span class="o">[</span><span class="n">20:49:06.918</span><span class="o">]</span><span class="w"> </span><span class="n">TPS</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">injection</span><span class="w"> </span><span class="p">(</span><span class="n">target</span><span class="p">)</span><span class="err">:</span><span class="w"> </span><span class="mi">1000</span>
<span class="o">[</span><span class="n">20:49:06.918</span><span class="o">]</span><span class="w"> </span><span class="n">TPS</span><span class="w"> </span><span class="k">of</span><span class="w"> </span><span class="n">injection</span><span class="w"> </span><span class="p">(</span><span class="n">de</span><span class="w"> </span><span class="n">facto</span><span class="p">)</span><span class="err">:</span><span class="w"> </span><span class="mf">180.96</span>
<span class="o">[</span><span class="n">20:49:06.918</span><span class="o">]</span><span class="w"> </span><span class="n">Empirical</span><span class="w"> </span><span class="nl">TPS</span><span class="p">:</span><span class="w"> </span><span class="mf">180.07</span>
<span class="o">[</span><span class="n">20:49:06.962</span><span class="o">]</span><span class="w"> </span><span class="o">[</span><span class="n">SUCCESS</span><span class="o">]</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="o">/</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="n">tezos_tps_benchmark</span>
</code></pre></div>
<p>We can see that on my laptop the empirical <span class="caps">TPS</span> result for the minimal
network is 180.</p>
<p>The goal of the <span class="caps">TPS</span> benchmark is to give a high-level estimate of the <span class="caps">TPS</span>
value that the system is capable of. It can be used to catch <span class="caps">TPS</span>
regressions, but not to find where exactly the bottlenecks are.</p>
<p>The empirical <span class="caps">TPS</span> is significantly affected by the hardware on which the
benchmark is run and other factors, such as the amount of logging that is
performed. For example, passing <code>-v</code> is likely to result in lower empirical
<span class="caps">TPS</span> values. This is why it is important to run this kind of benchmark on a
dedicated runner that has predictable performance.</p>
<p>The empirical <span class="caps">TPS</span> should normally be very close to the de facto <span class="caps">TPS</span> of
injection. If it isn’t, then it means that the system cannot keep up with
the injection rate, i.e. the bottleneck is in the system. Otherwise the
bottleneck is in the injecting code, as is the case in the log above.</p>
<h2 id="automatic-daily-runs-of-the-tps-benchmark"><a class="toclink" href="#automatic-daily-runs-of-the-tps-benchmark">Automatic daily runs of the <span class="caps">TPS</span> benchmark</a></h2>
<p>Every day the <span class="caps">TPS</span> benchmark is run and the following results are registered:</p>
<ul>
<li>Gas <span class="caps">TPS</span></li>
<li>Results of running the <span class="caps">TPS</span> benchmark with protocol limits:</li>
<li>De facto <span class="caps">TPS</span> of injection</li>
<li>Empirical <span class="caps">TPS</span></li>
<li>Results of running the <span class="caps">TPS</span> benchmark with protocol limits lifted:</li>
<li>De facto <span class="caps">TPS</span> of injection</li>
<li>Empirical <span class="caps">TPS</span></li>
</ul>
<p>Regressions for these values are detected and recorded using the same
framework as the one Tezt long tests use.</p>
<h2 id="current-limitations-and-future-work"><a class="toclink" href="#current-limitations-and-future-work">Current limitations and future work</a></h2>
<p>Since every smart contract is unique, we cannot automatically provide
support for all smart contracts that might become popular in the future.
Therefore, some work will be necessary in order to add more smart contracts
to the benchmark.</p>
<p>Another area for improvement is making the injecting code more efficient so
that it can reach higher rates of injection.</p>Announcing Tezos’ 10th protocol upgrade proposal “Jakarta”2022-04-16T12:00:00+02:002022-04-16T12:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-04-16:/announcing-tezos-10th-protocol-upgrade-proposal-jakarta.html<p>Announcing our next Tezos protocol proposal upgrade: Jakarta.</p><p><em>This is a joint post from Nomadic Labs, Marigold, TriliTech, Oxhead
Alpha, Tarides, DaiLambda, Functori <span class="amp">&</span> Tweag.</em></p>
<p><strong>Update: A revised prootocol proposal, <a href="https://research-development.nomadic-labs.com/announcing-jakarta-two.html">Jakarta
2</a>,
has been released. Jakarta 2 adresses <a href="https://research-development.nomadic-labs.com/we-found-two-bugs-in-torus-jakarta.html">two critical
bugs</a>
in the implementation of Transaction Optimistic Rollups.</strong></p>
<p>We were proud to see <a href="https://research-development.nomadic-labs.com/ithaca-is-live.html">Ithaca 2 go
live</a>
on April 1st. In keeping with <a href="https://research-development.nomadic-labs.com/regular-scheduling-for-our-tezos-proposals.html">our policy of proposing
upgrades</a>
on a regularly scheduled basis, we are happy to announce our latest
Tezos protocol proposal, Jakarta.</p>
<p>Jakarta’s “true name” is its hash,
<code>PtJakartaiDz69SfDDLXJSiuZqTSeSKRDbKVZC8MNzJnvRjvnGw</code>.</p>
<!—
As usual, Jakarta’s “true name” is its hash — which will be revealed
soon after the beginning of the [next election
cycle](https://tzstats.com/election/39).
—->
<p>The Jakarta protocol proposal contains major updates to the Tezos
economic protocol, as well as numerous minor improvements. In this
article, we preview its most relevant features. A more thorough
description can be read in the protocol proposal’s <a href="https://tezos.gitlab.io/jakarta/protocol.html">technical
documentation entry</a>,
and a complete list of changes is provided in its
<a href="https://tezos.gitlab.io/protocols/013_jakarta.html">changelog</a>.</p>
<h3 id="transaction-optimistic-rollups"><a class="toclink" href="#transaction-optimistic-rollups">Transaction optimistic rollups</a></h3>
<p>We are pleased to announce that <a href="https://tezos.gitlab.io/jakarta/transaction_rollups.html">Transaction Optimistic
Rollups</a> (or
TORUs) have found their way into this protocol proposal, as a first
and experimental implementation of optimistic rollups on Tezos. As the
name implies, TORUs allow for exchanges of assets, but not execution
of smart contracts.</p>
<p>Optimistic rollups enable higher throughput (<span class="caps">TPS</span>) by moving the
validation of transactions away from the main chain, to ‘Layer
2’. They are called <em>optimistic</em>, because they work on the assumption
that this validation is correct until explicitly proven otherwise. One
consequence of this approach is longer finality when withdrawing from
the rollup, currently set to 40,000 blocks (about 2 weeks).</p>
<p>Optimistic rollups have a number of desirable properties:</p>
<ul>
<li><strong>Trust minimized</strong>: You don’t have to trust that a majority of the
rollup nodes are honest to always be able to withdraw your funds
from the rollup. One honest node is enough.</li>
<li><strong>Permissionless</strong>: Anyone can submit operations to a rollup since
all the rollup block data is posted on the main chain.</li>
<li><strong>Capital efficient</strong>: Unlike with state channels (e.g., Lightning
Network), rollup users are not required to lock up a bond
upfront. Only rollup node providers are.</li>
</ul>
<p>With our proposal, Tezos rollups will not be implemented as smart
contracts (like, e.g., Arbitrum on Ethereum), but rather natively in
the economic protocol — making them <em>enshrined</em> rollups. Leveraging
Tezos’ unique self-amendment feature this way allows us to implement
more efficient, expressive, and scalable solutions.</p>
<p>For a broader understanding of optimistic rollups, and the rationale
behind them, we recommend checking out our blog post outlining <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html">a
scaling strategy for
Tezos</a>.</p>
<p><em>Note that TORUs should be considered an experimental feature, and
they are introduced with a sunset of 1 year. This sunset can be
extended, or removed altogether, in future proposals, depending on
community adoption.</em></p>
<h3 id="a-safer-sapling-integration"><a class="toclink" href="#a-safer-sapling-integration">A safer Sapling integration</a></h3>
<p>We recently <a href="https://research-development.nomadic-labs.com/fixing-the-sapling-protocol-integration.html">reported a design
flaw</a>
of the existing <a href="https://tezos.gitlab.io/ithaca/sapling.html">integration of Sapling
transactions</a> into
Michelson smart contracts, which makes <em>unshielding</em> tez from certain
smart contracts vulnerable to manipulation.</p>
<p>The Jakarta protocol addresses this situation by implementing a new,
safer design.</p>
<p>Jakarta also deprecates the previous format of the Michelson
integration for Sapling transactions, and if Jakarta is adopted, it
will only be possible to originate Sapling smart contracts which
conform to the new version.</p>
<p>Further details on the vulnerability and the new integration design
can be found
<a href="https://research-development.nomadic-labs.com/fixing-the-sapling-protocol-integration.html">here</a>.</p>
<h3 id="liquidity-baking-toggle-vote"><a class="toclink" href="#liquidity-baking-toggle-vote">Liquidity Baking toggle vote</a></h3>
<p>The Liquidity Baking Escape Hatch mechanism has been redesigned and
renamed to the <a href="https://tezos.gitlab.io/jakarta/liquidity_baking.html#toggle-vote"><em>“Liquidity Baking Toggle
Vote”</em></a>.</p>
<p>The options are now <strong>“On”</strong>, to vote for the Liquidity Baking subsidy
being turned on, and <strong>“Off”</strong>, to vote for the subsidy to be turned
off, and a new <strong>“Pass”</strong> option to abstain.</p>
<p>Another change is that if the threshold for deactivation is reached,
the deactivation of the subsidy is no longer permanent. If the
proportion of bakers voting “On” later increases back over the
threshold, the subsidy can be restarted.</p>
<p>More information can be found in the feature’s
<a href="https://gitlab.com/tezos/tzip/-/blob/master/drafts/current/draft-symmetric-liquidity-baking-toggle-vote.md"><span class="caps">TZIP</span></a>.</p>
<p><em>Note that the future Tezos Octez v13 baking daemon for Jakarta,
<code>tezos-baker-013-PtJakart</code>, will impose the use of a mandatory
command-line flag <code>--liquidity-baking-toggle-vote</code> to set up the
delegate’s default Liquidity Baking toggle vote preference, and will
also provide an optional <code>--votefile</code> flag intended to be used to
declare the path for a <span class="caps">JSON</span> file encoding the delegate’s toggle
vote. When provided, the latter file takes precedence over the former
mandatory flag.</em></p>
<h3 id="michelson-interpreter-improvements"><a class="toclink" href="#michelson-interpreter-improvements">Michelson interpreter improvements</a></h3>
<p>We have implemented various improvements to type safety and
performance of the Michelson interpreter. The majority of these
changes do not impact the semantics of Michelson other than decreasing
gas costs for parsing and unparsing scripts.</p>
<p>The single semantic change is ignoring annotations. With Jakarta,
annotations are only used by the type-checker and the interpreter to
identify smart contract entry-points. Additional annotations still
remain valid but they no longer carry any semantic meaning.</p>
<p>Moreover, the protocol proposal upgrades a few smart contracts which
relied on legacy features which are no longer available since the
Babylon protocol upgrade. They are now compliant with the <a href="https://tezos.gitlab.io/active/michelson.html">modern
Michelson
specification</a>. The
list of patched contracts, the patches themselves, and a detailed
description of the process can be found at the <a href="https://gitlab.com/tezos/tezos/-/merge_requests/3730">merge
request</a>.</p>
<h3 id="tickets-hardening"><a class="toclink" href="#tickets-hardening">Tickets hardening</a></h3>
<p><a href="https://tezos.gitlab.io/michelson-reference/#type-ticket">Tickets</a>
provide a first-class notion of ownership in the Tezos Protocol. They
can be used to implement fungible as well as non-fungible tokens.</p>
<p>We introduce a mechanism for explicitly tracking ownership of tickets
in the protocol. Whenever a ticket is created or its ownership changes
— for instance by sending it to a different contract — it is
explicitly recorded and validated against a balance table. This
extension does not impact the Michelson <span class="caps">API</span>.</p>
<p>The feature serves two purposes:</p>
<ol>
<li>
<p>Extra protection against attempts to forge tickets</p>
</li>
<li>
<p>Facilitate Layer 2 solutions that use tickets to represent assets
that can be exchanged with the main chain (e.g.,
<a href="https://tezos.gitlab.io/jakarta/transaction_rollups.html#workflow-overview">TORUs</a>)</p>
</li>
</ol>
<h3 id="rolls-are-no-more"><a class="toclink" href="#rolls-are-no-more">Rolls are no more</a></h3>
<p>The Jakarta protocol proposal redefines the computation of delegates’
<a href="https://tezos.gitlab.io/jakarta/voting.html#voting-power">voting
power</a> in
the self-amendment process. Instead of being measured in terms of
<a href="https://tezos.gitlab.io/active/voting.html#voting-power"><em>rolls</em></a>, it
is now defined directly by delegate’s stake (expressed in mutez). The
minimal stake required to be assigned voting rights is kept at 6000 tez.</p>
<p>This change complements <a href="https://research-development.nomadic-labs.com/tenderbake-has-been-injected.html#rewards">those introduced with the Ithaca2
protocol</a>,
in order to unify voting and baking power. As result the notion of
rolls is no longer relevant, and has been deprecated.</p>
<h2 id="testing-testing"><a class="toclink" href="#testing-testing">Testing, testing…</a></h2>
<p>A testnet for the Jakarta protocol named Jakartanet will launch in the
coming days. It is critical to have as many bakers, tool builders,
indexers, wallets, etc. as possible participating in this testnet.</p>
<p>We are looking for more bootstrap bakers to participate from day
one. If you are interested, please <a href="https://tezos-kiln.org/joinbakingslack">join the Tezos baking
slack</a> and reach out in the
<code>#test-networks</code> channel.</p>
<p>Furthermore, we strongly encourage you to test your own Tezos-based
applications for compatibility problems with Jakarta. Jakarta, and the
configuration for its test network Jakartanet, will be included in
version 13 of Octez.</p>
<p>Should the Jakarta protocol proposal be accepted by the community, v13
of Octez or v3 of TezEdge will be necessary to participate in
consensus due to necessary changes introduced to the protocol environment.</p>
<p>Over the coming months, our teams will continue to work on increasing
performance, lowering gas consumption, reducing block times, and
increasing the overall throughput — as measured, for example, in
transactions per seconds or smart contract invocations per second. We
are excited to be part of this continued development of Tezos.</p>Activating Tenderbake — a story in data2022-04-13T18:00:00+02:002022-04-13T18:00:00+02:00Nomadic Labs and Functoritag:research-development.nomadic-labs.com,2022-04-13:/activating-tenderbake.html<p>We take a look at the first cycle of Tenderbake to evaluate the performance of the network, and how it affected the bakers rewards</p><!-- Spell check: en US -->
<p>On April 1st, 2022, the Tezos blockchain successfully executed its
most ambitious protocol upgrade to date. At block
<a href="https://tzstats.com/2244609">#2,244,609</a>, the
<a href="https://research-development.nomadic-labs.com/ithaca-is-live.html">Ithaca2</a>
protocol upgrade was activated on Tezos mainnet, moving the Tezos
network away from the
<a href="https://research-development.nomadic-labs.com/emmy-seven-years-of-updatable-consensus.html">Emmy</a>
family of consensus algorithms to
<a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">Tenderbake</a>.</p>
<p>It is not the first time we have made changes to the consensus
algorithm via the on-chain amendment process, but this time we have
collectively gone where nobody in this space has gone before:
hot-swapping to a completely different consensus algorithm on a live
network without a hard-fork in just 1815 seconds.</p>
<p>Yes, it took only ~30 minutes for a majority of the network
participants to surf the migration waters, get the first block
endorsed by 2/3 of the network, and deliver Odysseus home to Ithaca.</p>
<p>We have been monitoring the network closely since then, and we are
glad to see that the transition from Emmy* to Tenderbake was generally
frictionless, and that the new consensus algorithm is running without
interruptions since the first block of the cycle.</p>
<!-- ~~[^friction]: There have been some minor hiccups reported, and
some have already been fixed with a new Octez release. However, there
are a few lingering issues. We are working on cleaning up those
smaller issues - and we will discuss them later. However, we do not
want these to let us lose focus on the fact that Tenderbake deployment
has been overall a great success!~~ -->
<p>In this article, we present some early observations and reflections on
the behavior of the network during<a href="https://tzstats.com/cycle/468"> cycle
468</a> - that is, the first 8192 blocks
of the Ithaca2 protocol, between Friday evening and Monday evening
(Paris time).</p>
<p>The data for this article was retrieved from the<a href="https://api.tzkt.io/"> TzKT
<span class="caps">API</span></a>.</p>
<h2 id="say-hello-to-fast-deterministic-finality"><a class="toclink" href="#say-hello-to-fast-deterministic-finality">Say hello to fast deterministic finality!</a></h2>
<p>The following table discriminates the first 8192 blocks of cycle 468,
from the activation of Ithaca2 on block
<a href="https://tzstats.com/2244609">#2,244,609</a> until block
<a href="https://tzstats.com/2260992">#2,260,992</a> on <span class="caps">CEST</span> Monday, April 4th.</p>
<figure align="center">
<img src="/images/ithaca-stats/stats_table.png" alt="Cycle 468 overall statistics" width="90%">
<figcaption align="center"> <b>Table 1:</b> Cycle 468 statistics by block round
</figcaption>
</figure>
<p>Consensus on the head of the chain was both <em>safe</em> and <em>fast</em>: the
average time between blocks was <strong>32.058s</strong>, close to the theoretical
minimum of 30 seconds. This is a direct consequence of having an
overwhelming majority of blocks proposed and agreed upon in round 0:
7936 blocks out of 8192 – around 97% of them.</p>
<p>The <a href="https://tezos.gitlab.io/alpha/consensus.html#overview"><em>round
number</em></a> for a
block denotes the number of attempts (minus one) that were necessary
to reach a consensus on a block at a given level. Lower is better, and
a high number of rounds indicates a slow or unreliable network. Under
normal network conditions, the chain should reach a consensus about
the next block in round 0 – that is, on the first attempt.</p>
<p>Of the remaining 256 blocks, 236 were produced on round 1, 11 on round
2, and 5 blocks in rounds between 3 and 5. The only two outliers were
understandably the first two blocks of the protocol migration, which
passed respectively on rounds 14 and 12. The latter were, also
understandably, the slower blocks of the cycle, with a time between
blocks of <strong>1815s </strong>and <strong>1590s</strong>.</p>
<p>These facts are better illustrated observing the values for these
attributes throughout the whole cycle. <a href="#fig-2">Figure 2</a> plots, for
each blockchain level, the <em>payload</em> and <em>block</em> rounds. Further
below, <a href="#fig-3">Figure 3</a> plots the <em>time between blocks</em> without
outliers above round 3<sup id="fnref:nicely"><a class="footnote-ref" href="#fn:nicely">1</a></sup>.</p>
<figure align="center" id="fig-2">
<img src="/images/ithaca-stats/block_payload_round_468.png" alt="Cycle 468 block/payload rounds per level" width="100%">
<figcaption align="center"> <b>Figure 2:</b> Block and payload proposal rounds per level in cycle 468
</figcaption>
</figure>
<p>With Tenderbake, we distinguish between the <em>payload</em> producer and the
<em>block</em> proposer. The payload is the non-consensus content of a block
(transfers, smart contract calls, etc.), and <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html#first-round-vs-subsequent-rounds">it can
be</a>
<em>locked</em> even if there was no consensus reached at the round. A baker
with rights to propose a block on a higher round must then re-use the
same payload – in jargon, it can <em>re-propose</em> the payload. Thus the
<em>payload round</em> is the round at which the block’s payload has been
first proposed (at the current level), and it is smaller or equal to
the block round.</p>
<p>In the majority of cases, the payload and block rounds match, even for
blocks proposed at rounds higher than 0. Even when Tenderbake <a href="https://research-development.nomadic-labs.com/tenderbake-has-been-injected.html">splits
baking
rewards</a>
(awarding 10 ꜩ to the payload producer and a variable bonus to the
block producer), in practice, most blocks in the cycle have the whole
of the baking rewards allocated to the same baker. This also
illustrates that even for higher rounds, the payload was first
proposed at the same round as that of the block that was finalized,
entailing that the failure to achieve a quorum on the first round was
more likely to be caused by absent bakers in the committee, rather
than a slow or untimely propagation of preendorsements.</p>
<figure align="center" id="fig-3">
<img src="/images/ithaca-stats/time_between_blocks_linear_468.png" alt="Cycle 468 overall statistics" width="100%">
<figcaption align="center"> <b>Figure 3:</b> Time between blocks per level in cycle 468
</figcaption>
</figure>
<p>As we mentioned above, the average time between blocks was
<strong>32.058s</strong>, pushed downwards by the majority of round 0 blocks. For
round 1 blocks, it was <strong>62.521s</strong>, or a bit over a minute. In <a href="fig-3">Figure
3</a>, we also observe a significant minority of blocks with a
<strong>45s</strong> time between blocks. This corresponds to round 0 blocks which
follow a round 1 block, as the<a href="https://tezos.gitlab.io/active/consensus.html#time-between-blocks"> time between
blocks</a>
measures the difference between two consecutive blocks’ timestamps,
and the timestamps are set at the beginning of the round. Thus, the
difference in such cases matches the length of the duration of round 1
– exactly 45s. These corner cases also explain why the average values
for the time between blocks at a given round presented in the table
above, e.g., for round 0, diverge slightly from the theoretical
minima. It moreover explains the maximal outliers, e.g., the <strong>90s</strong>
time between blocks for <a href="https://tzstats.com/2250528">#2,250,528</a>. The
latter is a round 0 block which follows a round 4 predecessor,
<a href="https://tzstats.com/2250527">#2,250,527</a>.</p>
<h2 id="a-safe-network-means-higher-bonuses"><a class="toclink" href="#a-safe-network-means-higher-bonuses">A safe network means higher bonuses</a></h2>
<p>Starting with Tenderbake,<a href="https://research-development.nomadic-labs.com/tenderbake-has-been-injected.html#the-block-its-payload-and-baking-rewards"> a part of the baking rewards of up to 10
ꜩis paid to the
baker</a>
for having included in a block extra endorsements for its predecessor
resulting in more than the minimal 4667 endorsements slots. That is,
beyond the 2/3rds of the total validations required to keep the chain
<em>live</em>. On average, the blocks in the first cycle had 6795.5 endorsed
slots out of 7000 — or a 97%. This not only resulted in the majority
of fast round 0 blocks we described before, but was also reflected in
the bonus rewards paid to the bakers: the average bonus included in
each block was <strong>9.125202 ꜩ.</strong></p>
<p><a href="#fig-4">Figure 4</a> plots the distribution of bonus baking rewards for
each level of cycle 468: the first few blocks included lower bonuses,
as the network was still recovering from the protocol migration and
some bakers were catching up. In fact, the bonus rewards averaged
<strong>4.227153 ꜩ</strong> in the first 100 blocks, but then we see how it rose
quickly to stabilize over 8 ꜩ by the 1000th block. Past the
quarter-cycle mark, it settled over 9 ꜩ. In fact, the average bonus
rewards for the trailing three-quarters of the cycle was <strong>9.379600
ꜩ</strong>.</p>
<figure align="center" id="fig-4">
<img src="/images/ithaca-stats/baker_bonus_468.png" alt="Cycle 468 baker bonus" width="100%">
<figcaption align="center"> <b>Figure 4:</b> Bonus rewards awarded per level in cycle 468
</figcaption>
</figure>
<p>Notice that around level
<a href="https://tzkt.io/2250609/operations">#2250609</a>, we can observe a
noticeable drop in the bonus reward: this is consistent with a large
drop in validation power due to a top public baker with a significant
stake being out temporarily. This highlights the fact that given that
Tenderbake favors <em>safety</em> over <em>liveness</em>, ensuring that the network
is live — and that everybody reaps the rewards — relies more than
ever on the collective effort.</p>
<h2 id="the-sweeping-crew-is-still-working"><a class="toclink" href="#the-sweeping-crew-is-still-working">The sweeping crew is still working</a></h2>
<p>In the days following the activation, a few bakers using Ledger
signers have been reporting issues with their baker software: the
Ledger seems to either occasionally freeze and disconnect when signing
(pre)endorsements, or return parsing errors when signing operations or blocks.</p>
<p>These can cause the baking daemon to freeze for up to ~10 minutes,
resulting in endorsement misses and occasionally lost baking
slots. This is happening seldomly, and we estimate it will not affect
the participation of the affected bakers enough to make them miss the
full endorsement rewards for the cycle. Sadly, this is not the case
for the occasional blocks lost.</p>
<p>We have been keeping busy with these (<a href="https://forum.tezosagora.org/t/releasing-octez-v12-3/4448">and other minor
issues</a>),
which should be fully addressed by an upcoming release of the Ledger
Tezos baking app.</p>
<h2 id="i-am-hooked-on-tenderbake-give-me-more-to-read"><a class="toclink" href="#i-am-hooked-on-tenderbake-give-me-more-to-read">I am hooked on Tenderbake, give me more to read</a></h2>
<p>Along the ride to deliver Tenderbake to Tezos, we have produced and
shared reports on the development of Tenderbake and its features. If
you want to read more about Tenderbake, we leave you a few good reads:</p>
<p>several blog entries;</p>
<ul>
<li><a href="https://research-development.nomadic-labs.com/tenderbake-has-been-injected.html">https://research-development.nomadic-labs.com/tenderbake-has-been-injected.html</a></li>
<li><a href="https://functori.com/blog/blog-tenderbake-baker-as-state-machine.html">https://functori.com/blog/blog-tenderbake-baker-as-state-machine.html</a></li>
<li><a href="https://research-development.nomadic-labs.com/simulating-tenderbake.html">https://research-development.nomadic-labs.com/simulating-tenderbake.html</a></li>
<li><a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html</a></li>
</ul>
<p>a few academic articles;</p>
<ul>
<li>L. Aştefănoaei, P. Chambart, A. Del Pozzo, T. Rieutord, S. Tucci, E. Zălinescu. <a href="https://arxiv.org/abs/2001.11965">Tenderbake — A Solution to Dynamic Repeated Consensus for Blockchains</a>. In 4th International Symposium on Foundations and Applications of Blockchain (<span class="caps">SCFAB</span>) 2021.</li>
<li>S. Conchon, A. Korneva, Ç. Bozman, M. Iguernlala, A. Mebsout. <a href="https://doi.org/10.4230/OASIcs.FMBC.2021.4">Formally Documenting Tenderbake</a>. In 3rd Workshop on Formal Methods for Blockchains (<span class="caps">FMBC</span>‘21), July 2021.</li>
</ul>
<p>and, of course, the main documentation entry-point:</p>
<ul>
<li><a href="https://tezos.gitlab.io/ithaca/consensus.html">https://tezos.gitlab.io/ithaca/consensus.html</a></li>
</ul>
<p>If you made this far, you might be interested to know that our
<a href="https://www.nomadic-labs.com/careers/distributed-consensus-algorithm-expert/">consensus team is
hiring</a>!</p>
<h2 id="lessons-and-looking-ahead"><a class="toclink" href="#lessons-and-looking-ahead">Lessons and looking ahead</a></h2>
<p>The activation of the Ithaca2 protocol marks the end of a
two-years-plus-long journey undertaken by the research and development
teams of Nomadic Labs, Functori, Tweag, and other Tezos core
developers and academic partners, like <span class="caps">CEA</span> List and Université Paris-Saclay.</p>
<p>The superlative performance of Tenderbake so far is also a consequence
of the preparedness of the community. A large majority of bakers were
ready for Tenderbake, and had done their homework in time. We are also
thankful to the many community members that helped out spreading the
word, and were ready to help others in different spaces. After all,
this is a decentralized, collective effort.</p>
<p>Tenderbake is alive and Odysseus is finally home on Ithaca, but it
does not mean we are done. We <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html">have ambitious
plans</a>
for the future of the Tezos blockchain, and the performance of the
early Tenderbake cycles gives us confidence to keep pushing the boundaries.</p>
<p>Sails are going to be raised back up soon.</p>
<p>Fair winds!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:nicely">
<p>Otherwise, <em>it does not plot nicely</em>, and we cannot
distinguish between blocks on rounds 0, 1, and 2. <a class="footnote-backref" href="#fnref:nicely" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Ithaca 2, the latest Tezos upgrade, is LIVE!2022-04-01T21:00:00+02:002022-04-01T21:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-04-01:/ithaca-is-live.html<p>Tezos has activated the ninth upgrade to the protocol, Ithaca 2. It replaces the current consensus algorithm (Emmy*) with Tenderbake – a revolutionary upgrade that sets the stage for a new era of innovation for the Tezos protocol.</p><p>On 1 April 2022, the Tezos blockchain successfully upgraded by
adopting the <a href="https://tezos.gitlab.io/protocols/012_ithaca.html">Ithaca
2</a> proposal at
block <a href="https://tezblock.io/block/2244609">#2,244,609</a>.</p>
<p>It is the ninth Tezos protocol upgrade and was jointly developed by
<em>Nomadic Labs, Marigold, TriliTech, Oxhead Alpha, Tarides, DaiLambda,
Functori <span class="amp">&</span> Tweag</em>.</p>
<p>Ithaca 2 introduces Tenderbake, a new consensus algorithm for Tezos. A
comprehensive introduction can be found
<a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">here</a>,
but the headlines are:</p>
<ul>
<li>
<p><strong>Fast, deterministic finality</strong>: A block can be trusted to be
final after two blocks, regardless of delays in network
communication. Under normal circumstances this is one minute.</p>
</li>
<li>
<p><strong>Choosing a safe network over a live network</strong>: As long as 2/3 of
the stake is honest, Tenderbake allows no parallel block production
that can potentially revert transactions. Any small fork stops
producing blocks, and if more than 1/3 of the total stake is isolated,
the entire network will halt and await reconnection.</p>
</li>
<li>
<p><strong>The road to faster blocks</strong>: While block time is kept at 30
seconds in Ithaca 2, Tenderbake makes it safe to introduce lower block
times in future protocol upgrades.</p>
</li>
</ul>
<p>A number of other changes are mostly relevant for bakers:</p>
<ul>
<li>
<p><strong>Lower minimum stake</strong>: The minimum amount of tez required to
receive baking/endorsement rights is reduced from 8,000 to 6,000.</p>
</li>
<li>
<p><strong>Instant rewards</strong>: Baking/endorsement rewards will no longer be frozen
for 5 cycles as has been the case up until now.</p>
</li>
<li>
<p><strong>Fewer problems with overdelegation</strong>: Instead of making a deposit
with each baked/endorsed block, bakers freeze 10% of their stake
upfront in each cycle. Bakers can set a limit for how much of their
stake will be used. This prevents missed slots from overdelegation.</p>
</li>
<li>
<p><strong>Total stake is used</strong>: Rolls are no longer used for assigning
baking/endorsement rights. Bakers will receive slots in proportion to
their actual stake, which should benefit small bakers in particular.</p>
</li>
<li>
<p><strong>More endorsements, steady participation</strong>: Endorsement slots per
block are increased from 256 to 7,000. This means a baker with the
minimum amount of tokens will participate every 10 blocks on average.</p>
</li>
</ul>
<h2 id="also-included-in-ithaca-2"><a class="toclink" href="#also-included-in-ithaca-2">Also included in Ithaca 2</a></h2>
<ul>
<li>
<p><strong>Precheck of manager operations</strong>: The new version of the protocol
allows any Tezos <em>shell</em> (e.g., Octez and TezEdge) to avoid fully
executing manager operations (those created by end users,
e.g. transfers and smart contract calls) before gossiping them through
the network. This lets operations reach bakers faster, and is a
prequel to further optimizations that can increase throughput.</p>
</li>
<li>
<p><strong>Liquidity Baking</strong>: Ithaca 2 includes an increase to the
liquidity baking sunset level of 819,200 blocks, or twenty voting
periods, roughly an additional ten months. To balance this
increase, the threshold for activating the escape hatch is lowered
from 50% to 33%.</p>
</li>
</ul>
<p><strong>… and more:</strong> We invite you to look at the
<a href="https://tezos.gitlab.io/protocols/012_ithaca.html">changelog</a> for a
full description of the contents of Ithaca 2, which includes <a href="https://tezos.gitlab.io/protocols/012_ithaca.html#new-environment-version-v4">a new
environment version
(V4)</a>,
<a href="https://tezos.gitlab.io/protocols/012_ithaca.html#michelson">updates to
Michelson</a>,
as well as <a href="https://tezos.gitlab.io/protocols/012_ithaca.html#minor-changes">minor
changes</a>
and <a href="https://tezos.gitlab.io/protocols/012_ithaca.html#bug-fixes">bug
fixes</a>.</p>
<p>Congratulations to everyone involved in the development of this
protocol amendment and welcome to the Tezos blockchain, Ithaca 2!</p>
<p>A special thank you to the Tezos community for updating nodes, bakers,
signer software/hardware, indexers, etc., in <a href="https://research-development.nomadic-labs.com/all-hands-on-deck-for-tenderbake.html">preparation for
Tenderbake</a>
— and for reaching out to others to make sure everyone was ready.</p>
<p>Our next protocol proposal, “J”, is targeted for injection in
mid-April. It will be the first of a series of <a href="https://research-development.nomadic-labs.com/tezos-is-scaling.html">scaling-focused
upgrade</a>
proposals meant to prepare Tezos for high-throughput use cases and
long-term growth. Stay tuned!</p>Get ready to roll. Tezos is scaling2022-03-28T17:00:00+02:002022-03-28T17:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2022-03-28:/tezos-is-scaling.html<p>A series of Tezos protocol proposals in 2022 will focus on paving the way for further adoption, high-throughput use cases and expanded smart contract functionality.</p><!-- Cover: https://research-development.nomadic-labs.com/images/tezos-scaling_lowres.jpg -->
<p><strong><span class="caps">TL</span>;<span class="caps">DR</span>: A series of Tezos protocol proposals in 2022 will focus on paving the way for further adoption, high-throughput use cases and expanded smart contract functionality.</strong></p>
<p>In case you haven’t heard: 2022 is the year of scaling for Tezos.</p>
<p>The new <a href="https://research-development.nomadic-labs.com/tenderbake-has-been-injected.html">Tenderbake</a> consensus mechanism enables lower block times, which will improve <em>latency</em> and <em>finality</em> – how quickly a transaction is included and made irreversible by the network. For the user it means faster transactions and smoother running dapps.</p>
<p>But Tenderbake doesn’t significantly change how many transactions can be processed per second, <em>throughput</em>. And that is the essence of scaling: enabling more people to do more on the network at the same time.</p>
<p>As increased adoption brings more projects, companies, and institutions into the Tezos ecosystem, it’s time to prepare the network for more activity and high-throughput applications.</p>
<p>However, there are natural limits to how much we can increase throughput on the main chain, <em>Layer 1</em>. We are pursuing every possible avenue to push the limits but going beyond them has a cost. Any blockchain doing so sacrifices either decentralization, latency, stability, security, censorship resistance, or a mix of these.</p>
<p>If we want to avoid this for Tezos, we need a different approach. To support very high-throughput use cases and prepare Tezos for long-term growth, we therefore turn to <em>optimistic rollups</em>, a <em>Layer 2</em> solution.</p>
<p>It’s a solution that can be implemented now to fulfill Tezos’ short-term scaling needs. We expect optimistic rollups to initially offer a 10-100x increase in throughput. And with further upgrades already in the making, it’s a step towards scaling on a whole new level.</p>
<h2 id="the-best-of-both-worlds"><a class="toclink" href="#the-best-of-both-worlds">The best of both worlds</a></h2>
<p>Scaling fundamentally comes in two flavors.</p>
<p>You can use more powerful computers able to process and store more transactions, <em>vertical scaling</em>, or you can split the workload across a higher number of computers, <em>horizontal scaling</em>. Both have benefits and drawbacks for blockchains.</p>
<p>Pure <em>vertical scaling</em> – requiring powerful hardware to participate on the main chain – is a straightforward solution, but it means fewer people are able to participate. The security and censorship resistance of the main chain depends on an honest majority of block producers along with other nodes validating the chain, so it’s important to have low barriers for participation.</p>
<p>Some degree of vertical scaling can be achieved through software optimizations, allowing more throughput with the same hardware. This is a constant focus for Tezos development and has already resulted in significant performance improvements on the main chain – for example by <a href="https://research-development.nomadic-labs.com/delphi-the-latest-tezos-upgrade-is-live.html">refining the gas model</a>, <a href="https://research-development.nomadic-labs.com/granada-the-latest-tezos-upgrade-is-live.html">optimizing the Michelson interpreter</a>, and <a href="https://www.marigold.dev/post/hangzhou-the-latest-tezos-upgrade-is-live">introducing a cache for smart contracts</a>.</p>
<p><em>Horizontal scaling</em> of blockchains is mostly done by splitting up the main chain workload across clusters of nodes, known as <em>sharding</em>. It preserves decentralization, but it’s a complex solution and interaction between clusters adds latency – something we have just sought to minimize on the main chain with Tenderbake.</p>
<p>Optimistic rollups are a way to <a href="https://www.youtube.com/watch?v=oqBSs0DSuzQ">take elements from both approaches</a> and apply them where it makes the most sense.</p>
<p>Large batches of transactions are moved from the main chain to rollup-specific nodes, a form of horizontal scaling. These transactions can then be processed using more powerful hardware, achieving vertical scaling.</p>
<p>Meanwhile, the security of the main chain acts as a backstop in case of foul play.</p>
<h2 id="rollups-in-a-nutshell"><a class="toclink" href="#rollups-in-a-nutshell">Rollups in a nutshell</a></h2>
<p>The technical details of optimistic rollups on Tezos will be covered in upcoming blog posts, but the fundamental principle is this:</p>
<p>A rollup is an entity on the main chain, with its own address, that compactly represents off-chain transaction execution and state updates. Assets can be deposited to the rollup – and withdrawn – by accounts, smart contracts and other rollups.</p>
<p>Transactions using the rollup are still sent to an <em>inbox</em> on the main chain, but are left unprocessed by the main chain nodes. Instead they are collected by specialized <em>rollup operators</em>.</p>
<p>The operators, who store and maintain the <em>state</em> of a rollup, process the transactions and update the state to reflect the changes. This all happens off-chain, on <em>Layer 2</em>. The operator only posts a receipt back to the main chain, summarizing the new state of the rollup as a cryptographic hash. A <em>commitment</em>.</p>
<p>This process of continuously “rolling up” transactions into commitments is where the name comes from. Collect, update, commit, repeat. The cycle is executed as fast as possible by the rollup operators, using powerful hardware if needed.</p>
<p>By contrast, main chain nodes simply store incoming rollup transaction data and the commitments. They are relieved of the computationally expensive task of validating the transactions, which lets them produce and validate main chain blocks at a higher rate.</p>
<p>The system is <em>optimistic</em> because a commitment is treated as being correct unless someone disputes it. If no dispute happens within a given time period, assets can be withdrawn back to the main chain.</p>
<p>To incentivize honest behavior by the rollup operators, there are economic penalties for posting incorrect commitments. Should a dispute happen, the main chain serves as a court.</p>
<p>This doesn’t just make incorrect commitments costly for the operator: any incorrect commitment is ultimately neutralized. As long as there is just <em>a single honest node</em> checking commitments, and the main chain is censorship resistant, posting incorrect commitments is futile.</p>
<h2 id="a-rollup-roadmap"><a class="toclink" href="#a-rollup-roadmap">A rollup roadmap</a></h2>
<p>Optimistic rollups are implemented in line with the general Tezos approach: gradual evolution through incremental upgrades.</p>
<p>The development is a joint effort by teams at Nomadic Labs, Marigold, Oxhead Alpha, Functori, Tarides, DaiLambda, and TriliTech.</p>
<p>The roll-out will differentiate between <em>transaction rollups</em> and <em>smart contract rollups</em>.</p>
<p>Both are implemented as protocol upgrades rather than deployed as smart contracts, which allows for more gas- and storage-efficient implementations. They are also permissionless. Anyone will be able to launch a rollup or be an operator for any rollup.</p>
<ul>
<li>
<p><strong>Transaction rollups</strong> are expected to be part of the upcoming “J” upgrade proposal. They can handle account-to-account transactions of assets, but won’t be able to run smart contracts. This allows for a simpler design and faster implementation.</p>
</li>
<li>
<p><strong>Smart contract rollups</strong> are targeted for later this year. The chosen approach is inspired by proven designs for rollups, but tailored to Tezos. They are designed with a generic structure that is not limited to Michelson smart contracts, but can be adapted to support environments like WebAssembly or <span class="caps">EVM</span>, the Ethereum Virtual Machine.</p>
</li>
</ul>
<h2 id="next-level-scaling"><a class="toclink" href="#next-level-scaling">Next-level scaling</a></h2>
<p>While optimistic rollups are a good scaling solution for Tezos, they do currently have limitations.</p>
<p>One issue is that all transaction data for the rollup inbox must be included in blocks on the main chain, but there is a limit to how much data can be stored in each block.</p>
<p>Therefore, avenues for further scaling are being researched and developed.</p>
<p>While sharding <em>all</em> main chain functionality isn’t an approach we’re currently pursuing for Tezos, sharding <em>can</em> be used to increase the size of the inbox for rollups.</p>
<p>The approach is called <em>data-availability sharding</em>, and has the major benefit of further increasing rollup throughput as more bakers join the network. Horizontal scaling at work.</p>
<p>Instead of having every main chain node download and store all incoming rollup transaction data, the data is spread among the network nodes, and among bakers in proportion to their stake. Essentially making bakers data providers for rollup nodes.</p>
<p>This increases the total bandwidth and capacity, which improves throughput for rollups significantly. If rollups alone offer a 10-100x increase in throughput, the increase is 1,000-10,000x when combined with data-availability sharding.</p>
<h2 id="the-zero-knowledge-track"><a class="toclink" href="#the-zero-knowledge-track">The zero-knowledge track</a></h2>
<p>Another issue is the time period needed for commitments to be checked and potential disputes resolved.</p>
<p>Sending assets to a rollup and transacting inside it has the same finality as the main chain. But to withdraw assets back to the main chain, you must wait for the dispute period to pass. On Arbitrum, a popular rollup system on Ethereum, the period is currently seven days.</p>
<p>In parallel to the work of implementing optimistic rollups, we are therefore also exploring <em>zero-knowledge rollups</em>, or <em>zk-rollups</em>.</p>
<p>The principle of moving transactions off-chain and posting commitments to the main chain is the same as with optimistic rollups, but zk-rollups attach a cryptographic proof of correctness with the commitment. No need for a waiting period for withdrawals.</p>
<p>This is possible because <em>zk-proofs</em> are very small and can be checked by main chain nodes in a fraction of a second, regardless of the amount of transactions or smart contract operations involved.</p>
<p>The downside is that zk-proofs are currently highly time consuming to <em>generate</em>. It requires a disproportionate amount of computing power, especially when dealing with smart contracts. The zero-knowledge team at Nomadic Labs is however working on improving this.</p>
<p>We look forward to updating the community, as the work on data-availability sharding and zk-rollups progresses.</p>
<p>In the meantime, let’s get ready to roll.</p>We Discovered a Flaw in the Sapling Protocol Integration — a fix is ready for J.2022-03-15T16:00:00+01:002022-03-15T16:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2022-03-15:/fixing-the-sapling-protocol-integration.html<p>A design flaw in the <code>SAPLING_VERIFY_UPDATE</code> Michelson opcode renders <em>unshielding</em> transactions from the shielded pools implemented in <code>sapling_contract.tz</code> malleable. This affects any deployment of this sample contract on the Tezos mainnet, and could potentially extend to other contracts following a similar pattern. We have implemented a new, strengthened, version of this primitive which should become the required opcode for safer integration of Sapling transactions in Michelson smart contracts. This new version will be available as a part of the upcoming J protocol proposal. In the meantime, we advise against originating new Sapling contracts on mainnet, and to avoid interacting with the existing contracts.</p><!--
[GAD: Beginning of the rendered content]
[GAD: a bit long for a TL;DR now. Renaming to a summary]
-->
<p><strong>Summary: A design flaw in the <code>SAPLING_VERIFY_UPDATE</code> Michelson opcode renders <em>unshielding</em> transactions from the shielded pools implemented in <code>sapling_contract.tz</code> malleable. This affects any deployment of this sample contract on the Tezos mainnet, and could potentially extend to other contracts following a similar pattern. We have implemented a new, strengthened, version of this primitive which should become the required opcode for safer integration of Sapling transactions in Michelson smart contracts. This new version will be available as a part of the upcoming J protocol proposal. In the meantime, we advise against originating new Sapling contracts on mainnet, and to avoid interacting with the existing contracts.</strong></p>
<p>This blog post follows a discussion in <a href="https://forum.tezosagora.org/t/sapling-test-contract-is-it-safe-audited-ready-for-mainnet/3446">Tezos Agora</a>, where a vulnerability is revealed in <a href="https://gitlab.com/tezos/tezos/-/blob/ebetanet-release/src/proto_008_PsZ2k9Fh/lib_protocol/test/contracts/sapling_contract.tz">sapling_contract.tz</a>, a sample Michelson smart contract distributed as a part of the test suite of the Tezos protocol. This flaw could potentially apply to other contracts implementing a similar pattern.</p>
<p>The bug in the contracts arises from a design flaw in one of the Michelson opcodes provided to integrate Sapling transactions to Tezos smart contracts, <a href="https://tezos.gitlab.io/012/michelson.html#sapling-operations"><code>SAPLING_VERIFY_UPDATE</code></a>. That is, this is not a vulnerability in the Sapling protocol, its implementation, or even the implementation of the Michelson opcode <em>per se</em>. Instead, it is rather the result of its specification, and the provided interface, not being sufficiently strong to write safe smart contracts.</p>
<!--- [AB: it's a bug fix for the contract, but it's a design fix for the instruction. The instruction itself isn't buggy or unsafe, it's just not what is needed to write a safe contract] [GAD: check] -->
<p>Over the last few days, we have implemented a patch addressing this issue on <a href="https://gitlab.com/tezos/tezos/-/merge_requests/4589">tezos!4589</a>, which will be included with the next protocol proposal for <a href="https://gitlab.com/tezos/tezos/-/milestones/56">protocol J</a>, strengthening the support for shielded transactions.</p>
<p><strong>In the meantime we advise the Tezos community to avoid originating new contracts creating shielded pools, and to avoid interacting with the deployed contracts. Even if we have not observed any exploit of this vulnerability, we are confident an attack is indeed plausible.</strong></p>
<!-- We acknowledge this is unquestionably a critical issue in the Tezos protocol. Still, a recent inspection of deployed contracts on the Tezos mainnet at [level 2155024](https://tzstats.com/2155024), reveals only 2 contracts originated using the sample vulnerable code from `sapling_contract.tz`[^data]. At the time of writing, the outstanding balance in the two shielded pools is under 6 tez. -->
<p>A recent inspection of deployed contracts on the Tezos mainnet at <a href="https://tzstats.com/2155024">level 2155024</a>, reveals only 2 contracts originated using a copy of the vulnerable shielded pool implemented by <code>sapling_contract.tz</code>: <a href="https://tzkt.io/KT1BUZy6xNbmo5ogfD7nDRMBspfG1EEFfDFW/operations/">KT1BUZy6x</a> and <a href="https://tzkt.io/KT1E3kx2Wc7qoSfR4zZmTMA7P84nQnKENwW5">KT1E3kx2W</a><sup id="fnref:data"><a class="footnote-ref" href="#fn:data">1</a></sup>. At the time of writing, the outstanding balance in the two shielded pools is under 6 tez. We recommend users to avoid interacting with them.</p>
<p>Given the low exposure in locked liquidity in the shielded pool, as this feature has not been used extensively on mainnet so far, we consider that shipping this fix with the J protocol proposal is the best alternative among other available options.</p>
<!---
[GAD: That said. Do we want to mention explicitly the Airgap contract? add a link to an explorer? They already tweeted about this
https://twitter.com/AirGap_it/status/1498703941617475588
so this is public information.
If so, do still we need to check with them first that they are OK with the content? ]
We have witnessed several unshield operations [following Airgap's announcement on Twitter](https://twitter.com/AirGap_it/status/1498703941617475588).
-->
<p>In the sequel, we provide further detail on the vulnerability and we describe the efforts undertaken to lift the current limitations of the Sapling protocol integration.</p>
<h2 id="sapling-and-unshielding-transactions"><a class="toclink" href="#sapling-and-unshielding-transactions">Sapling and unshielding transactions</a></h2>
<!-- Explain the basics of sapling, point to posts about Edo.-->
<p>Since the activation of the <a href="https://research-development.nomadic-labs.com/edo-the-latest-tezos-upgrade-is-live.html">Edo2 protocol proposal</a> on block <a href="https://tzstats.com/1343489">1,343,489</a> a little over a year ago, the Tezos economic protocol has provided integration for a <a href="https://tezos.gitlab.io/012/sapling.html">Sapling protocol</a>, which enables privacy-preserving transactions. The design and implementation of Sapling in Tezos follows closely the original design by the Electric Coin Company for <a href="https://z.cash/">ZCash</a>. <a href="https://tezos.gitlab.io/011/sapling.html#tezos-integration">Its integration</a> into the Tezos economic protocol provides also two Michelson opcodes, <code>SAPLING_EMPTY_STATE</code> and <code>SAPLING_VERIFY_UPDATE</code>, to enable application developers to seamlessly integrate Sapling transactions into their smart contracts. This opened the door to new type of privacy-preserving applications, in particular asset transactions with selective disclosure — as implemented in <a href="https://gitlab.com/tezos/tezos/-/blob/ebetanet-release/src/proto_008_PsZ2k9Fh/lib_protocol/test/contracts/sapling_contract.tz">sapling_contract.tz</a>.</p>
<!-- Detail the current design -->
<p>In order to support such applications, the <a href="https://tezos.gitlab.io/011/michelson.html#sapling-operations"><code>SAPLING_VERIFY_UPDATE</code></a> provides a unified interface for both: <em>shielding</em> (i.e., depositing) tez into a shielded pool, <em>unshielding</em> (i.e., withdrawing) tez from a pool, and transferring shielded tokens between shielded accounts — all this using a single interface. The specification reads:</p>
<p><code>SAPLING_VERIFY_UPDATE / t : s : S => Some (Pair b s') : S</code></p>
<p>In words, it prescribes that the opcode consumes a shielded transaction <code>t</code> and a Sapling state <code>s</code> from the top of the execution stack. It will then verify the transaction and, if it succeeds, it returns <code>Some</code> outstanding balance <code>b</code>, and a new state Sapling state <code>s'</code><sup id="fnref:none"><a class="footnote-ref" href="#fn:none">2</a></sup>. The balance <code>b</code> is a signed integer value which determines an outstanding balance that needs to be consolidated with the smart contract’s unshielded balance:</p>
<ul>
<li>A strictly-positive balance value entails <em>unshielding</em> tokens — e.g., to match burned shielded tokens in a pool.</li>
<li>A strictly-negative balance value entails <em>shielding</em> tez — e.g., to compensate minted shielded tokens in a pool.</li>
<li>A zero-valued balance denotes an internal shielded operation which doesn’t prescribe a change in the unshielded balance of the contract — e.g. an internal transfer between shielded accounts.</li>
</ul>
<p>Thus, in the case of non-zero balances, a strictly-positive <code>b</code> entails that <code>b</code> tez need to be debited from the contract to restore the application’s invariant. Respectively, a strictly-negative <code>b</code> entails that <code>b</code> tez have to be credited to the contract to preserve the invariant.</p>
<!---
> [GAD: please check I got the semantics right.]
> [GAD: Marcin. Does this clarify that _balance_ is the outstanding one? — MP: yes, definitely. GAD: Thanks!]
--->
<p>The flaw in the current integration lies within the interaction between the Sapling protocol and the Tezos protocol, in particular when withdrawing tokens from a shielded pool. As a general rule, when designing an operation that transfers ownership of a token, it is crucial that the key owning the tokens signs a permission to debit a certain amount, <em>together with the address of the recipient</em>. If the recipient is not signed by the owner, then an attacker can intercept the transaction, and swap the recipient with its own address. For example in the case of a shielding transaction, the Tezos account that owns the tez signs a transfer which includes the Sapling recipient. Unfortunately, the converse is not true. In the case of an unshield, there is no way for a Sapling key to sign the Tezos recipient, given the current format of Sapling operations as introduced in Edo.</p>
<p>In short: when someone withdraws tez from the shielded pool, an attacker observing the pending smart contract call can <em>steal</em> the Sapling transaction parameter, and use it to inject a new smart contract call, with their own address as recipient. This can lead to a loss of funds, but it doesn’t compromise the privacy of the shielded pool.</p>
<!--- Detail the potential vulnerability in the contract -->
<p>Now, in order to illustrate this design flaw, we take a closer look at <a href="https://gitlab.com/tezos/tezos/-/blob/ebetanet-release/src/proto_008_PsZ2k9Fh/lib_protocol/test/contracts/sapling_contract.tz">sapling_contract.tz</a>. This contract manages a shielded pool where tokens are pegged 1 to 1 to tez. This tokens can be shielded to and unshielded from the pool, or transferred between shielded accounts, using Sapling transactions. We focus on a few key selected lines.</p>
<p>On line 5, we observe the parameter declaration. The first parameter is the Sapling transaction to be verified by the protocol, denoted by <code>(sapling_transaction 8)</code> type — just ignore the 8 value there<sup id="fnref:why"><a class="footnote-ref" href="#fn:why">3</a></sup>. The second has a <code>key hash</code> option type:</p>
<!--- **[TODO: what's 8? the size? in bytes? nat? MB : It's the size of the memo, and encrypted arbitrary bytes in each output. The protocol doesn't look at it.]** -->
<div class="highlight"><pre><span></span><code><span class="k">parameter</span><span class="w"> </span><span class="p">(</span><span class="kt">list</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">sapling_transaction</span><span class="w"> </span><span class="m">8</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">option</span><span class="w"> </span><span class="kt">key_hash</span><span class="p">)</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="p">);</span>
</code></pre></div>
<!--- Marco's original
The latter is an optional implicit account, which denotes the intended destination, when _unshielding_ the tokens from the pool. As you can see the two arguments are given separately and there is no cryptographic signature binding the two. In other words the combination of Tezos smart contract call and Sapling unshield is *malleable*. An attacker can thus intercept an unshield operation destinated to a Tezos address and submit it to the contract with its own address, effectively stealing the unshielded tez. Let's see why:
---->
<p>The latter is an optional implicit account, which denotes the intended destination, when <em>unshielding</em> the tokens from the pool. As you can see, the two arguments are given separately and there is no cryptographic signature binding the two. In other words, the combination of Tezos smart contract call and Sapling unshield is <em>malleable</em>. An attacker can intercept an unshield operation, and proceed to resubmit it with its own address as destination instead, effectively stealing the unshielded tez. Let’s see why:</p>
<!--- MS we can remove from here --->
<p>The whole interaction with the Sapling protocol on this contract, as we hinted before, is concentrated on the <code>SAPLING_VERIFY_UPDATE</code> opcode on line 21:</p>
<div class="highlight"><pre><span></span><code><span class="c1"># We verify the transaction and update the storage if the transaction is</span>
<span class="c1"># valid. The shielded transactions are handled here.</span>
<span class="c1"># The new state is pushed on top of the stack in addition to the balance</span>
<span class="c1"># of the transaction. If the rest of the script goes well, this state</span>
<span class="c1"># will be the new state of the smart contract.</span>
<span class="w"> </span><span class="kr">SAPLING_VERIFY_UPDATE</span><span class="p">;</span>
</code></pre></div>
<p>The remark in the docstrings are worth revisiting: the pushed state will be the new state of the smart contract, making proof replay impossible. In the case of an <em>unshield</em> — a positive balance being pushed to the stack as a result — , the interesting bit of the contract’s effect takes place jumping forward to the <code>IFGT</code> opcode expanding over lines 36 — 45:</p>
<div class="highlight"><pre><span></span><code><span class="kr">IFGT</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kr">DIIP</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">ASSERT_SOME</span><span class="p">;</span>
<span class="w"> </span><span class="kr">IMPLICIT_ACCOUNT</span><span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">UNIT</span><span class="p">;</span>
<span class="w"> </span><span class="kr">TRANSFER_TOKENS</span><span class="p">;</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="c1"># Stack manipulation to order.</span>
<span class="w"> </span><span class="c1"># The operations will consist of the# TRANSFER_TOKEN operation.</span>
<span class="w"> </span><span class="kr">DIP</span><span class="w"> </span><span class="p">{</span><span class="kr">CONS</span><span class="p">}</span><span class="w"> </span><span class="p">;};</span>
</code></pre></div>
<p>If on line 36 the value on the top of the stack is greater than zero, then, on line 41, we will <code>TRANSFER_TOKENS</code> (that is, unshielded tez) to the address pushed on the stack to restore the 1 to 1 peg, in this case the second parameter provided to the contract.</p>
<!-- Explain the attack surface. -->
<p>Now, we see clearly that there is no <em>binding</em> of the destination consumed as an argument by <code>TRANSFER_TOKENS</code>, and the Sapling transaction verified by the protocol, as we mentioned above. Enforcing the operational correctness of <em>unshield</em>, that is, ensuring that tokens arrive to their intended destination, is a burden of the rest of the smart contract — and ultimately, its developer. This contract does not implement any such mechanism, and therefore is indeed <em>malleable</em>. An adversary could <em>plausibly</em> exploit this by <em>observing</em> a valid call to this contract, and subsequently replace the destination with the attacker’s public key hash.</p>
<!--- MS we can remove to here --->
<!-- Ponder on this bug
>@knayawp: Maybe say that this contract was just an example, not intended/ready for production. It might be a bit weaselly.
-->
<p>Indeed, this is a critical bug in the contract, and any other contract following this pattern on mainnet is vulnerable as well. This is clearly an unintended design error on the integration of the Sapling protocol, and not in the implementation of the Michelson <span class="caps">API</span> — which is correct with regard to its specification — nor, on the Sapling protocol — which is still, safe and sound.</p>
<p>That said, the limitations of the current existing <span class="caps">API</span> are clear, and the Tezos protocol needs an improved set of primitives for safe, privacy-preserving transactions. We discuss our immediate first step in the next section.</p>
<!---
[AB: there was clearly an unintended design error, but one that a review of the contract did catch, the issue is not in the implementation of the instruction itself, but in the way it needs to be used] [GAD: check]
--->
<h2 id="strengthening-shielded-transactions"><a class="toclink" href="#strengthening-shielded-transactions">Strengthening shielded transactions</a></h2>
<p>In order to address this issue, we have <a href="https://gitlab.com/tezos/tezos/-/merge_requests/4589">implemented</a> the following changes to the Sapling integration in Tezos and Michelson, which will be included in the upcoming <a href="https://gitlab.com/tezos/tezos/-/milestones/56#tab-issues">J protocol proposal</a>. Adopting the proposed changes will prevent the issues described above, and enable Tezos smart contract developers to write safer privacy-preserving applications. In further detail:</p>
<ul>
<li>
<p>We add an extra field to Sapling transactions, called <code>bound_data</code>. This field is <strong>signed by the Sapling spending key</strong>, as a part of the whole transaction. In the case of an unshield, it is intended to contain its recipient.</p>
</li>
<li>
<p>We update the Michelson type <code>sapling_transaction</code>, and we overload and extend the <a href="https://tezos.gitlab.io/active/michelson.html#sapling-operations"><code>SAPLING_VERIFY_UPDATE</code></a> opcode to handle the updated transaction type. When provided the new transaction type, <code>SAPLING_VERIFY_UPDATE</code> performs the same checks done by the previous version, and it additionally checks the signature on the supplied <code>bound_data</code> field. The latter is returned together with the balance and the updated state.</p>
</li>
<li>
<p>We provide a new <code>sapling_contract.tz</code> reference smart contract which expects the Tezos address of the recipient to be included in the <code>bound_data</code> field of the Sapling transaction. Moreover, the address should be encoded as a Micheline <code>public_key_hash</code> type. <strong>Thus, instead of having the recipient as a separate parameter to the smart contract, it is extracted from the Sapling transaction</strong>.</p>
</li>
<li>
<p>We update the Octez Tezos client to correctly populate the <code>bound_data</code> of a Sapling unshield transaction with the Tezos address of the recipient.</p>
</li>
<li>
<p>We deprecate the original Michelson type — which is also renamed to <code>sapling_transaction_deprecated</code>. As a result, it will continue to work on <code>mainnet</code> for previously originated contracts, but after (and if) protocol proposal J is activated, <strong>new smart contracts using the deprecated transaction type will no longer able to be originated</strong>, and the origination operation will fail.</p>
</li>
</ul>
<!--
[^other]: For _shielding_ transactions, this can be used for instance to include information for a third-party authentication system.
--->
<p>We are confident this strengthening will avoid similar issues arising from loose bindings between Sapling transactions, and their sources and destinations.</p>
<!-- - Confidence:
- Plans for increasing confidence in sapling?
- Tests? Can we have something for this, e.g. a Tezt which simulates the potential attack?
--->
<p><strong>It should be noticed that these changes will only be effective on the Tezos mainnet once — and only if — the J protocol proposal is accepted by the community, and after it is activated on the Tezos mainnet. Before then, we advise the community against deploying new contracts using the current support for Sapling transactions, and to avoid interacting with pre-existing deployed contracts. Shielded funds locked in the vulnerable contracts are safe, but attempting to <em>unshield</em> tokens from these pools entails a risk of being targeted.</strong></p>
<!--- next steps and mitigation efforts --->
<!---
That said, if any user still wants to _unshield_ their remaining shielded tez, injecting the contract call directly to the public node of a trusted baker would minimize the risk of an attack.
GAD: Removing this as most users have already unshielded.
-->
<h2 id="looking-ahead"><a class="toclink" href="#looking-ahead">Looking ahead</a></h2>
<!--
- We are not clearly happy about this situation.
- Thank the community again. Highlight we proposed a quick fix.
- Revisit why we are deciding to fix this for J, instead of patching I2 during the next vote.
- Invite people to join the testnets - and experiment with these contracts.
--->
<p>It is regretful that this critical issue with the integration of the Sapling protocol exists, as we were very much looking forward to seeing increased adoption of privacy-preserving features. We are reevaluating our development process to make it harder and harder for such issues to slip through in the future and we renew our commitment to develop technologies that are innovative and trustworthy.</p>
<!--
The commitment to rely on well-tested, verifiable components is a core value of the Tezos ecosystem. It is also a genuine one, which makes it harder to be able to communicate about issues like this one.
--->
<p>If there is a silver lining, it is that this was caught by testing and reviewing a contract before it was massively adopted by the community. We are also reassured by the fact that a patch could be implemented quickly.</p>
<p>We invite the community, specially developers of applications building upon the Sapling protocol’s infrastructure, to test the new set of primitives on upcoming test networks. A teztnet for the J proposal will be announced soon, following the usual process for <a href="https://teztnets.xyz/">launching protocol test networks</a>. Another option is to join the rolling <a href="https://teztnets.xyz/dailynet-about">Dailynet</a> and <a href="https://teztnets.xyz/mondaynet-about">Mondaynet</a> teztnets — feel free to reach out to us if you need help deploying your contracts on the test networks.</p>
<!--
Feel free to reach out to us on the [Tezos baking Slack](https://tezos-baking.slack.com/) or to contact [the Nomadic Labs support team](https://adoption-support.nomadic-labs.com/contact) if you need help deploying your contracts on the test networks.
--->
<div class="footnote">
<hr>
<ol>
<li id="fn:data">
<p>We have observed three contracts which use the Sapling integration provided by <code>SAPLING_VERIFY_UPDATE</code>. In addition to the two contracts mentioned above, there is a third deployed contract, <a href="https://tzkt.io/KT1UmxfNXrKJvBDarGMiVA2Rnuaxn2Jb6sZQ">KT1UmxfNX</a>. This contract implements different functionality, it has not been used since its origination, and thus never held any tez balance. <a class="footnote-backref" href="#fnref:data" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:none">
<p>We elide here the failing cases which would push <code>None</code> to the stack, as they are not needed to illustrate the bug. <a class="footnote-backref" href="#fnref:none" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:why">
<p>The value denotes the size of the <a href="https://tezos.gitlab.io/012/sapling.html#memo"><em>memo</em></a>, encrypted arbitrary bytes attached to each output, and only accessible to the owner of the viewer/spending keys. <a class="footnote-backref" href="#fnref:why" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>All Hands on Deck for Tenderbake2022-02-24T18:30:00+01:002022-02-24T18:30:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2022-02-24:/all-hands-on-deck-for-tenderbake.html<p>A successful transition to the Tenderbake consensus mechanism relies on the Tezos ecosystem making sure the infrastructure is prepared for the changes. Here is a checklist of the necessary steps.</p><!-- This article has been spelled-checked with an en_US dictionary -->
<p><strong>A successful transition to the Tenderbake consensus mechanism relies
on the Tezos ecosystem making sure the infrastructure is prepared for
the changes. Here is a checklist of the necessary steps.</strong></p>
<p>The <a href="https://research-development.nomadic-labs.com/announcing-ithaca-2.html">Ithaca2 protocol proposal</a> contains the perhaps most significant upgrade of
the Tezos protocol to date.</p>
<p>It introduces a new consensus mechanism,
<a href="https://arxiv.org/abs/2001.11965">Tenderbake</a>, which brings <a href="https://research-development.nomadic-labs.com/tenderbake-has-been-injected.html">several
improvements</a>
including deterministic finality, i.e. absolute certainty that
transactions cannot be reversed after two blocks.</p>
<p>Replacing the consensus mechanism is in itself a major undertaking, as
it defines the rules by which Tezos bakers decide the state of the
ledger – a core function of a blockchain.</p>
<p>Performing such an upgrade on a live, global blockchain network
further complicates things. It is essentially like replacing the
engine of a car while it is running, and it is important to note that
for certain types of existing infrastructure Tenderbake introduces
breaking changes.</p>
<p>A decentralized network is dependent on its constituent actors doing
their part, and in the Tezos ecosystem that means keeping on top of
necessary changes as the protocol evolves.</p>
<h2 id="it-can-get-bumpy"><a class="toclink" href="#it-can-get-bumpy">It can get bumpy</a></h2>
<p>After the Granada upgrade in August, the network experienced <a href="https://research-development.nomadic-labs.com/updates-granada.html">longer
block times and many missed
endorsements</a>,
temporarily lowering network <em>“health”</em>. After the Hangzhou upgrade in
November, <a href="https://www.marigold.dev/post/upcoming-activation-of-hangzhou">context
flattening</a>
caused some low-spec nodes to run out of memory.</p>
<p>In both cases the network stayed online and the issues were resolved,
but there is no denying that protocol upgrades can get bumpy.</p>
<p>Despite increased testing efforts on our part, it is not possible to
fully model the complexities of the mainnet, and this is where
preparation becomes important.</p>
<p>To increase the likelihood of a smooth transition, we call on all
ecosystem participants to make sure they are prepared for Ithaca2
activation, which will happen around March 31st, provided that the
upgrade is voted in by the community.</p>
<p>This includes bakers, block explorers, wallet providers, exchanges,
indexing service providers, node-as-a-service providers, dapp
maintainers, and everyone else involved in providing tooling or
services in the Tezos ecosystem.</p>
<h2 id="the-tenderbake-checklist"><a class="toclink" href="#the-tenderbake-checklist">The Tenderbake checklist</a></h2>
<p>Generally, we encourage the ecosystem to join the
<a href="https://teztnets.xyz/ithacanet-about"><code>ithacanet</code></a> testnet to make
sure their setup and infrastructure works with the upcoming protocol version.</p>
<p>As a minimum, the following should be completed by the time of activation:</p>
<ul>
<li>
<p><strong>Tezos node and baking software</strong> need to be updated to a
Tenderbake compatible version. For
<a href="https://gitlab.com/tezos/tezos/-/releases">Octez</a>, this is <code>v12.0</code>
and later versions. For <a href="https://github.com/tezedge/tezedge">TezEdge</a>,
it’s <code>v2</code> and later versions.</p>
</li>
<li>
<p><strong>Bakers using a Ledger hardware wallet</strong> for secure signing need
to update the Tezos Baking app on their device to <code>v2.2.15</code>. <strong>Earlier
versions will <span class="caps">NOT</span> work after Ithaca2 activation</strong>.</p>
</li>
<li>
<p><strong>Remote signing software for baking</strong> will need to be
significantly updated<sup id="fnref:remote"><a class="footnote-ref" href="#fn:remote">1</a></sup>.</p>
</li>
<li>
<p><strong>Block explorers and other indexing software</strong> will need to be
significantly updated.</p>
</li>
<li>
<p><strong>Dapp maintainers</strong> are highly encouraged to test their dapps on
the testnet.</p>
</li>
</ul>
<p>If you are unsure about what needs to be done on your end, reach out
to us on the <a href="https://tezos-baking.slack.com/">Tezos baking Slack</a> or
feel free to contact <a href="https://adoption-support.nomadic-labs.com/contact/">the Nomadic Labs support
team</a>.</p>
<h2 id="choosing-safety-over-liveness"><a class="toclink" href="#choosing-safety-over-liveness">Choosing safety over liveness</a></h2>
<p>Given the scope and complexity of the upcoming upgrade, we find it
important to remind the community that the Tenderbake consensus
mechanism marks a shift to <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">favoring the network being <em>safe</em> over
being
<em>live</em></a>.</p>
<p>The current <a href="https://research-development.nomadic-labs.com/emmy-seven-years-of-updatable-consensus.html">Emmy* consensus
mechanism</a>
allows for multiple versions of the network, forks, to run in parallel
during a major network split. This could be due to global internet
disruptions or a software bug. Similar to how Bitcoin and Ethereum work.</p>
<p>When connection between different forks is re-established, the version
with the largest stake (or for Proof-of-Work networks, the most
hash-power) will define the state of the ledger. Smaller forks are abandoned.</p>
<p>This keeps the network running, <em>live</em>, at all times, but carries the
possibility of transactions being reverted if they are on a smaller fork.</p>
<p>With Tenderbake this changes. As a so-called <a href="https://arxiv.org/abs/2001.11965"><em>classical <span class="caps">BFT</span>-style
consensus algorithm</em></a>, it operates
on the assumption that at least 2/3 of the total stake is honest.</p>
<p>As long as this is the case, there can be no parallel block production
that suddenly replaces or reverts transactions following a network
split. Any small fork will halt<sup id="fnref:fork"><a class="footnote-ref" href="#fn:fork">2</a></sup>. The network is
<em>safe</em>.</p>
<p>The trade-off is that if more than 1/3 of the total stake is isolated
or offline, the entire network will halt until
connection between at least 2/3 of the total stake is re-established,
rather than staying live as separate networks.</p>
<p>This is by design. Intended behavior. But it can be triggered by a
bug — or by a large enough number of unprepared network participants,
which this blog post aims to prevent.</p>
<h2 id="a-note-on-hardware"><a class="toclink" href="#a-note-on-hardware">A note on hardware</a></h2>
<p>As Tezos continues to gain adoption and will need to offer higher
throughput, questions have arisen whether hardware requirements for
bakers will increase.</p>
<p>We do not see the current upgrade to Tenderbake consensus making this
necessary. We expect current hardware setups to remain sufficient,
provided they live up to the <a href="https://wiki.tezosagora.org/use/baking/setting-up-a-secure-baker">generally recommended
specifications</a>.</p>
<p>Upcoming Layer 2 scaling initiatives, such as optimistic roll-ups, are
expected to greatly alleviate the pressure on the main chain, Layer 1.</p>
<p>However, Tenderbake is a first step towards tweaking the network to
increase throughput on Layer 1. It is possible that increased
adoption, resulting in fuller blocks, combined with tweaks of certain
network parameters can lead to the lowest-powered baking systems
eventually needing to be upgraded.</p>
<p>Should it become relevant, it is important to note that a slight
upgrade of the lowest-powered systems — e.g. from a Raspberry Pi to a
sufficiently powered Intel <span class="caps">NUC</span> — would not affect Tezos’ highly
energy-efficient profile in any significant way.</p>
<p>As the recent <a href="https://tezos.com/2021-12-06-Tezos-LCA-Final.pdf">PwC report assessing Tezos’ carbon
footprint</a> shows,
10% of respondent bakers reported using a Raspberry Pi, each drawing
about 9 watts. Switching to an Intel <span class="caps">NUC</span> would increase the energy
consumption of these bakers’ systems to about 12 watts.</p>
<p>Tenderbake is a major milestone for Tezos and its community. We are
excited to be a part of this major protocol transformation and eager
to see what new opportunities it will bring.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:remote">
<p>The change of consensus algorithm introduces new operations (pre-endorsements), new concepts (such as <em>rounds</em>), and changes the semantics of existing software components. This changes the byte payload of the consensus operations signed by bakers, and the rules to prevent double (pre)endorsing/baking. Hence, a new version of the signer software is required. Read more <a href="https://tezos.gitlab.io/protocols/tenderbake.html#signer">here</a>. <a class="footnote-backref" href="#fnref:remote" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:fork">
<p>Assuming the split is unintended. If more than 1/3 of the
total stake is held by bad actors intentionally performing a
coordinated attack, the network can indeed fork. <a class="footnote-backref" href="#fnref:fork" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>A POPL 2022 Retrospective2022-02-15T19:00:00+01:002022-02-15T19:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2022-02-15:/a-popl-2022-retrospective.html<p>A short summary of our experience at <span class="caps">POPL</span>, <span class="caps">CPP</span> and CoqPL this year.</p><p>A few weeks ago, we <em>physically</em> attended <a href="https://popl22.sigplan.org"><span class="caps">POPL</span> 2022</a>
— the 49th <span class="caps">ACM</span> <span class="caps">SIGPLAN</span> Symposium on Principles of Programming
Languages — which took place in Philadelphia, <span class="caps">PA</span> and
also <em>virtually</em> everywhere on Earth via
<a href="https://www.airmeet.com/e/83336870-5dde-11ec-82d0-a1d80a53071a">Airmeet</a>.</p>
<p><span class="caps">POPL</span> is a<sup id="fnref:the"><a class="footnote-ref" href="#fn:the">1</a></sup> premier annual conference event of the programming
languages research community showcasing, together with several
colocated events, the latest cutting edge results in programming
languages, especially in functional programming and formal
verification. The latter topics are very dear to the Tezos community,
as they enable us to build safety-critical, complex-yet-beautiful
software — such as the Tezos protocol and the
Octez suite — on sound technical foundations. Moreover, research
presented in conferences like <span class="caps">POPL</span> have immediate impact on our daily
work at Nomadic Labs.</p>
<p>This affinity is reflected in our continuous involvement as sponsors:</p>
<ul>
<li>
<p>The <a href="https://tezos.foundation/">Tezos Foundation</a> was a platinum
sponsor of <span class="caps">POPL</span> 2022, for the fourth consecutive year.</p>
</li>
<li>
<p>We at Nomadic Labs are pleased and proud to have sponsored the
colocated <a href="https://popl22.sigplan.org/home/CPP-2022"><span class="caps">CPP</span> 2022 (Conference on Certified Programs and
Proofs)</a>, for the third
consecutive year.</p>
</li>
</ul>
<p>After almost two years of virtual events, and debugging audio and video
connections across different virtualization platforms, we were eager
to return to in-person <span class="caps">POPL</span>-ing<sup id="fnref:2021"><a class="footnote-ref" href="#fn:2021">2</a></sup> and real-life <em>hallway</em>
tracks. Our on-site team consisted of: Michel Mauny, Nomadic Labs’
<span class="caps">CEO</span>; Germán Delbianco, Research Engineer at Nomadic Labs; and Michael
Holey, <a href="https://blokhaus.io/team/">Blokhaus’ community
manager</a><sup id="fnref:thanks"><a class="footnote-ref" href="#fn:thanks">3</a></sup>. They were present
throughout the week at the Tezos booth, eager and ready to answer
questions about <a href="https://tezos.com/">Tezos</a>. It was an amazing opportunity to promote our
interest in these research fields, and also to advertise possible
collaborations, openings <a href="https://tezos.com/careers/">in the Tezos ecosystem</a> and <a href="https://www.nomadic-labs.com/careers/">at Nomadic Labs</a>, as well as <a href="https://tezos.foundation/grants">research
funding opportunities</a>.</p>
<p>We are happy to have had the chance to catch up with the <span class="caps">PL</span> community,
and we hope that we have planted the seeds for fruitful collaborations
which will benefit the Tezos community. There was also plenty of
interesting, high-quality talks, and below we bring you a few (biased)
highlights of this year’s program<sup id="fnref:notes"><a class="footnote-ref" href="#fn:notes">4</a></sup>.</p>
<!-- What's a POPL and why we care -->
<h2 id="the-popl-program-and-our-highlights"><a class="toclink" href="#the-popl-program-and-our-highlights">The <span class="caps">POPL</span> Program and our Highlights</a></h2>
<p>Each year, the <span class="caps">ACM</span> <span class="caps">SIGPLAN</span> Symposium on Programming Languages — <span class="caps">POPL</span>
in short — features novel research contributions ranging from
theoretical foundations of programming languages (e.g. semantics and
type systems), to the development and application of formal tools for
crafting, and verifying, reliable and correct systems.</p>
<p>Among the other colocated events on offer during the week (like
<a href="https://popl22.sigplan.org/home/VMCAI-2022"><span class="caps">VMCAI</span></a>, or
<a href="https://popl22.sigplan.org/home/lafi-2022"><span class="caps">LAFI</span></a>), we took time to
attend <a href="https://popl22.sigplan.org/home/CPP-2022"><span class="caps">CPP</span> 2022</a> and <a href="https://popl22.sigplan.org/home/CoqPL-2022">CoqPL
2022</a>.</p>
<p>The full program is available
<a href="https://popl22.sigplan.org/program/program-POPL-2022/">here</a>, and
several events, including the main conference, have made the talks
available on <a href="https://www.youtube.com/channel/UCwG9512Wm7jSS6Iqshz4Dpg"><span class="caps">SIGPLAN</span>’s YouTube
channel</a>.</p>
<!--- POPL highlights -->
<h3 id="popl-2022"><a class="toclink" href="#popl-2022"><span class="caps">POPL</span> 2022</a></h3>
<p>This year the <span class="caps">POPL</span> program consisted of <a href="https://popl22.sigplan.org/track/POPL-2022-popl-research-papers#event-overview">60+ accepted
papers</a>,
in addition to 3 keynotes, and a special session of
<a href="https://dl.acm.org/journal/toplas"><span class="caps">TOPLAS</span></a> articles. There was plenty
of high-quality research catering to different tastes and appetites. An
incomplete list of those we enjoyed<sup id="fnref:par"><a class="footnote-ref" href="#fn:par">5</a></sup>:</p>
<ul>
<li>
<p><a href="http://www.cs.columbia.edu/~aho">Alfred Aho’s</a> keynote,
<a href="https://www.youtube.com/watch?v=SaiQdrprBFQ&list=PLyrlk8Xaylp4uPnEfcP5jETxhm50XTit9&index=67&ab_channel=ACMSIGPLAN"><em>“Principles of Programming Language
Translators”</em></a>
focused on the topic of <em>computational thinking</em>, and how it
applies to compiler design. In particular, it revisited what an
<em>abstraction</em> is, from a programming language perspective: we
should define an <em>abstraction</em> as a <em>model</em> and a set of operations
for manipulating data (the “programming language” of the data
model). The talk focused on a taxonomy of four abstractions for
computer science: fundamental abstractions, abstract
implementation, and declarative and computational abstractions; and
illustrated with examples from compiler design. Towards the end,
and moving away from compilers, Professor Aho focused on <em>quantum
computing</em>, and argued that the <em>four postulates of quantum mechanics</em><sup id="fnref:postulates"><a class="footnote-ref" href="#fn:postulates">6</a></sup>
define new kinds of abstractions that extend the previously
presented <em>classical</em> ones.</p>
</li>
<li>
<p><a href="https://www.youtube.com/watch?v=1m7ebAOH8CI&list=PLyrlk8Xaylp4uPnEfcP5jETxhm50XTit9&index=60&ab_channel=ACMSIGPLAN"><em><span class="dquo">“</span>Concurrent Incorrectness Separation
Logic”</em></a>. Incorrectness
separation logic has recently been introduced as a dual to
separation logic, whose goal is not to establish the correctness of
programs, but rather prove the presence of bugs, by catching them,
based on under-approximated reasoning. This work extends the logic
to a parametric framework to a concurrent, shared memory model,
which can be instantiated to soundly assess whether <em>races</em>,
<em>deadlocks</em>, and memory safety errors detected are true positives
— where “sound” means that all results are <em>true positives</em>.</p>
</li>
<li>
<p><a href="https://www.youtube.com/watch?v=XLblh1mKmP4&list=PLyrlk8Xaylp4uPnEfcP5jETxhm50XTit9&index=58&ab_channel=ACMSIGPLAN"><em><span class="dquo">“</span>The Leaky Semicolon: Compositional Semantic Dependencies for
Relaxed-Memory
Concurrency”</em></a>. This
talk featured a retrospective of 20 years of research in memory
models, looking into how to support sequential composition while
targeting modern hardware architectures, following an event-based
approach with preconditions and families of predicate transformers.</p>
</li>
<li>
<p><a href="https://www.youtube.com/watch?v=P4_FHYWnEzQ&list=PLyrlk8Xaylp4uPnEfcP5jETxhm50XTit9&index=61&ab_channel=ACMSIGPLAN">Simuliris: A Separation Logic Framework for Verifying Concurrent
Program
Optimizations</a>. This
talks presents <em>Simuliris</em>, a simulation technique to establish
termination preservation for a range of concurrent program
transformations that exploit <em>undefined behavior</em> in the source
language. The key idea is using <em>ownership</em> to reason modularly
about compiler assumptions about well-defined behavior. Based on
the <a href="https://iris-project.org/">Iris</a> framework.</p>
</li>
<li>
<p><a href="https://www.youtube.com/watch?v=ES1bhxcRDy0&list=PLyrlk8Xaylp4uPnEfcP5jETxhm50XTit9&index=24&ab_channel=ACMSIGPLAN"><em><span class="dquo">“</span>Pirouette: Higher Order Typed Functional
Choreographies”</em></a>. Pirouette
is a programming language for typed higher-order functional
distributed <em>choreography</em>. Pirouette offers programmers the ability
to write a centralized functional program and compile it into
programs for each node in a distributed system. The paper
provides in particular a formalization in Coq, notably showing that the soundness
of the type-system entails deadlock-freedom.</p>
</li>
<li>
<p>Last, we want to highlight an effort on smart contract
verification: <a href="https://www.youtube.com/watch?v=cS_yPCHbdY0&list=PLyrlk8Xaylp4uPnEfcP5jETxhm50XTit9&index=32&ab_channel=ACMSIGPLAN"><em>“SolType: Refinement Types for
Solidity”</em></a>. This
work presents a refinement type system for Solidity that can be
used to prevent arithmetic over- and under-flows in Ethereum smart contracts.</p>
</li>
</ul>
<p>This year, <span class="caps">POPL</span> also featured a virtual post-conference workshop
event, across multiple time zones. In addition to a few select invited
speakers, the novelty this year was a couple of speed-dating sessions
where industrial sponsors (like us) could connect with potential candidates.</p>
<p>All presentations are available on a dedicated <a href="https://www.youtube.com/playlist?list=PLyrlk8Xaylp4uPnEfcP5jETxhm50XTit9">YouTube
playlist</a>.</p>
<!--- CPP highlights -->
<h3 id="cpp-2022"><a class="toclink" href="#cpp-2022"><span class="caps">CPP</span> 2022</a></h3>
<p>As mentioned above, Nomadic Labs were proud sponsors of <a href="https://popl22.sigplan.org/home/CPP-2022"><span class="caps">CPP</span>
2022</a>, the 8th Conference on
Certified Programs and Proofs. This conference covers a broad spectrum
of mechanized verification efforts and tools, ranging from the
formalization of mathematics and certified algorithms to new proof
techniques, frameworks and tooling for interactive theorem proving.</p>
<p>This year, we highlight two very interesting keynotes that resonate
with our own verification endeavors at Nomadic Labs:</p>
<ul>
<li>
<p>Professor <a href="https://www.cs.princeton.edu/~appel/">Andrew Appel’s</a>
keynote, on <a href="https://www.youtube.com/watch?v=StQ40osfQTo&list=PLyrlk8Xaylp7peYx_GED0Ny3q-gf3fZYC&index=27&ab_channel=ACMSIGPLAN"><em>“Coq’s vibrant ecosystem for verification
engineering”</em></a>
addressed the engineering aspects of building vertical stacks
consisting of verified software components, centered around <a href="https://vst.cs.princeton.edu">the
<span class="caps">VST</span> project</a>, and how to scale up
verification efforts by relying on community libraries and
community-centric projects like <a href="https://github.com/coq/platform">the Coq
platform</a>. In particular he
focused on how to address the <em>verification gap</em> across multiple
layers of formalization efforts and mechanized software. The issue
of the verification gap, that is the <em>semantic</em> distance between
verification artifacts and <em>real-world</em> production code, is one of
the big hurdles of industrial scale projects like ours — and it
has for instance been at the center of different recent projects
and
<a href="https://research-development.nomadic-labs.com/three-questions-to-nomadic-labs-interns-antonio-locascio.html">internships</a>
in house.</p>
</li>
<li>
<p><a href="https://trustworthy.systems/people/?cn=June+Andronick">June
Andronick’s</a>
keynote, <a href="https://www.youtube.com/watch?v=QL5oGMxpmJI&list=PLyrlk8Xaylp7peYx_GED0Ny3q-gf3fZYC&index=22&ab_channel=ACMSIGPLAN"><em>“The seL4 verification: the art and craft of proof and
the reality of commercial
support”</em></a>,
consisted of two parts. The first one was a <em>tour de force</em> of the
<a href="https://sel4.systems">seL4</a> project, discussing in general the
scalability of large verification stacks, and in particular the
challenges of the project. The second part focused on the broader
picture of carrying out large scale verification projects in
industrial settings. She focused on both technical
<em>proof-engineering</em> aspects; as well as <em>non-technical</em> ones, like
how to <em>properly</em> explain the virtues of formal verification to
non-technical audiences. That rings a bell!</p>
</li>
</ul>
<p>Other talks we enjoyed were:</p>
<ul>
<li>
<p><a href="https://www.youtube.com/watch?v=3rkljJSEfrc&list=PLyrlk8Xaylp7peYx_GED0Ny3q-gf3fZYC&index=1&ab_channel=ACMSIGPLAN"><em><span class="dquo">“</span>Specification and Verification of a
Transient-Stack”</em></a>,
a very cool verification effort using
<a href="https://www.chargueraud.org/softs/cfml"><span class="caps">CFML</span></a> and <em>time credits</em>
in Coq to certify concurrent and persistent data structures.</p>
</li>
<li>
<p><a href="https://www.youtube.com/watch?v=mgun16f3C2k&list=PLyrlk8Xaylp7peYx_GED0Ny3q-gf3fZYC&index=2&ab_channel=ACMSIGPLA"><em><span class="dquo">“</span>Mechanized Verification of a Fine-Grained Concurrent Queue from Meta’s Folly Library”</em></a>, presents the formal specification and verification in <a href="https://iris-project.org/">the Iris framework</a> of a high-performance, fine-grained concurrent <a href="https://github.com/facebook/folly/blob/main/folly/MPMCQueue.h">multi-producer multi-consumer queue</a> from Meta’s <code>folly</code> library.</p>
</li>
</ul>
<p>Blockchain-related topics like certified frameworks for programming
correct smart contracts, or mechanized proofs of consensus algorithms,
frequently appear in the program. This year we had:</p>
<ul>
<li>
<p><a href="https://www.youtube.com/watch?v=rDt54nd3vUQ&list=PLyrlk8Xaylp7peYx_GED0Ny3q-gf3fZYC&index=24&ab_channel=ACMSIGPLAN"><em><span class="dquo">“</span>A verified algebraic representation of Cairo program
execution”</em></a>, which
focuses on how to verify <em>STARKs</em> in the
<a href="https://leanprover.github.io/"><span class="caps">LEAN</span></a> theorem
prover. <a href="https://starkware.co/starknet/">StarkNet’s</a> STARKs
constitute the core of a Zero Knowledge-based Layer 2 solution for
Ethereum — but eventually scalable to other blockchains — whose
aim is to translate <span class="caps">EVM</span> bytecode into simple, low level
<a href="https://www.cairo-lang.org/">Cairo</a> programs.</p>
</li>
<li>
<p><a href="https://www.youtube.com/watch?v=VwCBlmS7XEA&list=PLyrlk8Xaylp7peYx_GED0Ny3q-gf3fZYC&index=25&ab_channel=ACMSIGPLAN"><em><span class="dquo">“</span>Formal Verification of a Distributed Dynamic Reconfiguration
Protocol”</em></a>,
presenting the specification and formal verification using
<a href="https://lamport.azurewebsites.net/tla/tla.html"><span class="caps">TLA</span>+</a> of the core
<code>MongoRaftReconfig</code> reconfiguration protocol, implemented in the
<a href="https://en.wikipedia.org/wiki/MongoDB">MongoDB</a> distributed database.</p>
</li>
</ul>
<p>All <span class="caps">CPP</span> 2022 presentations are available on <span class="caps">ACM</span> <span class="caps">SIGPLAN</span>’s
YouTube channel
<a href="https://www.youtube.com/playlist?list=PLyrlk8Xaylp7peYx_GED0Ny3q-gf3fZYC">playlist</a>.</p>
<h3 id="coqpl-2022"><a class="toclink" href="#coqpl-2022">CoqPL 2022</a></h3>
<p>On the last day of the conference, we attended <a href="https://popl22.sigplan.org/home/CoqPL-2022">CoqPL
2022</a>, the Eighth
International Workshop on Coq for Programming Languages. The <a href="https://coq.inria.fr/">Coq
proof assistant</a> is one of Nomadic Labs’
favorite verification tools, as witnessed by <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/">the Mi-Cho-Coq
framework</a>, the
<a href="https://clarus.github.io/coq-of-ocaml/examples/tezos/">coq-of-ocaml</a>
project — and also,
<a href="https://www.nomadic-labs.com/careers/mechatez-formally-verifying-critical-features-of-tezos-protocols/">various</a>
<a href="https://www.nomadic-labs.com/careers/improve-our-formal-verification-framework">internship</a>
<a href="https://www.nomadic-labs.com/careers/fat-cat-formal-acceptance-testing-of-contracts-for-administering-tokens/">projects</a>
on offer. Moreover, the Tezos Foundation supports the development of
Coq and of its ecosystem (see <a href="https://tezos.foundation/grants">here</a>
how to apply for support), and the CoqPL workshop provides an
opportunity to interact with the Coq development team, learn about the
<a href="https://coq.discourse.group/t/coq-platform-2022-01-0/1545">recently released
features</a>
and what’s coming next in the pipeline, and other recent library developments.</p>
<!--- See you next year -->
<h2 id="see-you-next-year"><a class="toclink" href="#see-you-next-year">See you next year!</a></h2>
<p>It was a great experience for us to be back at <span class="caps">POPL</span>, and the hybrid
format allowed us to reach out and interact better with the community.</p>
<p><span class="caps">POPL</span> turns 50 next year, and it will again be a hybrid event, with the
physical event taking place somewhere in the United States’ West
Coast<sup id="fnref:ish"><a class="footnote-ref" href="#fn:ish">7</a></sup>. We look forwards to seeing you there for this special
occasion — both in person and online!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:the">
<p>or <em>the</em> premier conference, depending on whom you ask and how
many papers they got in this year. That said, it is indeed a very
reputable venue, and a mark of pedigree that can make a career as
a researcher in the field. <a class="footnote-backref" href="#fnref:the" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2021">
<p>That said, the still-ongoing pandemic did not stop us from
<a href="https://research-development.nomadic-labs.com/popl-2021-retrospective.html">attending (and blogging about) virtual <span class="caps">POPL</span> last
year</a>. <a class="footnote-backref" href="#fnref:2021" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:thanks">
<p>Many thanks also to Blokhaus’ Jenny Carbonaro who handled
logistics, and was in place to set up our cool stand, and welcome
our Nomads to Philly. <a class="footnote-backref" href="#fnref:thanks" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:notes">
<p>A more detailed (and even more biased) live blog by Germán
is available
<a href="https://codimd.nomadic-labs.com/s/_JbvpyfCP">here</a>. <a class="footnote-backref" href="#fnref:notes" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:par">
<p>A practical lesson in <em>linearizability</em>. In person, parallel
sessions, entail that you can enjoy <em>one</em> talk live, and
eventually watch the <em>other</em> (a few weeks) later on YouTube. This often
entails you find yourself in the wrong-room, or that there were
two talks <em>you really wanted to attend</em> scheduled at the same
time. You can attend virtual parallel sessions live in parallel,
at your own risk. <a class="footnote-backref" href="#fnref:par" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:postulates">
<p>Specifically, he referred to the four postulates of
quantum mechanics as they are presented in Chapter 2 of the
seminal <a href="https://en.wikipedia.org/wiki/Quantum_Computation_and_Quantum_Information"><em>“Quantum Computation and Quantum
Information”</em></a>
by Michael A. Nielsen <span class="amp">&</span> Isaac L. Chuang. <a class="footnote-backref" href="#fnref:postulates" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:ish">
<p>It has not being officially announced yet but preliminary
announcements hinted towards cities that, for instance, host teams
in the Western Conference of the <span class="caps">NBA</span>, regardless of what maps have
to say on the matter. <a class="footnote-backref" href="#fnref:ish" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
</ol>
</div>Refactoring the Management of Native Tokens in the Tezos Economic Protocol2022-02-01T14:00:00+01:002022-02-01T14:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2022-02-01:/refactoring-token-management.html<p>As part of our efforts to make the Tezos protocol easier to understand and safer to evolve, we have refactored the logic of <em>native tokens</em> management in the <strong>Ithaca</strong> protocol proposal.</p><p>Before the <a href="https://tezos.gitlab.io/012/protocol.html"><strong>Ithaca</strong></a> protocol proposal, <em>native protocol</em> token<sup id="fnref:native"><a class="footnote-ref" href="#fn:native">1</a></sup> (tez) movements and associated balance updates could be difficult to follow in the Octez source code for developers and maintainers.
For example, crediting or debiting a contract, or creating the corresponding balance updates, often happened at distant locations in the code, with a sometimes non-trivial control flow in between those operations.
With Ithaca, we have refactored the code to centralize token movements (e.g. transactions, fees payments, rewards, …) in a dedicated module, so that:</p>
<ol>
<li>Token transfers are more explicit and uniform, and associated balance updates are exhaustive.</li>
<li>It is easier to assess properties related to token movements, such as preserving the invariant <code>circulating tokens = minted tokens - burned tokens</code>. Here, the term <code>circulating tokens</code> refers to all tokens frozen or held by user and smart contracts.</li>
</ol>
<p>To achieve this, we aimed for an implementation of token transfers that is correct-by-construction in the following senses:</p>
<ul>
<li>Tokens are either minted and deposited into an account, moved from one account to another, or withdrawn from an account and burned; and</li>
<li>Balance updates found in block metadata give a complete and exact account of all tokens minted, moved or burned.</li>
</ul>
<p>The first property ensures that the total amount of tokens in circulation is equal to the difference between the amount of tokens minted and the amount of tokens burned.
Continuously ensuring theses properties while the protocol evolves can be particularly tedious when the implementation does not use an explicit notion of <em>token transfer</em> from one token holder to another.
The second property allows anyone to audit all token movements that happen when a block has been applied, just by looking at the balance updates in the block’s metadata.</p>
<h2 id="token-management-in-hangzhou"><a class="toclink" href="#token-management-in-hangzhou">Token management in Hangzhou</a></h2>
<p>In protocol <strong>Hangzhou</strong>, token transfers are implemented in a few steps consisting of debiting a sender of tokens, crediting a receiver of tokens and constructing balance updates to report the movement.
Even though those steps are semantically very close to each other, they are intermixed with other instructions and hence can be distant within the source code.
For example, a transfer from an implicit account to another is implemented as follows:</p>
<div class="highlight"><pre><span></span><code><span class="o">...</span>
<span class="nn">Contract</span><span class="p">.</span><span class="n">spend</span> <span class="n">ctxt</span> <span class="n">sender</span> <span class="n">amount</span> <span class="o">>>=?</span> <span class="k">fun</span> <span class="n">ctxt</span> <span class="o">-></span>
<span class="o">(</span><span class="k">match</span> <span class="nn">Contract</span><span class="p">.</span><span class="n">is_implicit</span> <span class="n">receiver</span> <span class="k">with</span>
<span class="o">|</span> <span class="nc">None</span> <span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="n">ctxt</span><span class="o">,</span> <span class="bp">[]</span><span class="o">,</span> <span class="bp">false</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">Some</span> <span class="o">_</span> <span class="o">-></span> <span class="o">(</span>
<span class="nn">Contract</span><span class="p">.</span><span class="n">allocated</span> <span class="n">ctxt</span> <span class="n">receiver</span> <span class="o">>>=?</span> <span class="k">function</span>
<span class="o">|</span> <span class="bp">true</span> <span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="n">ctxt</span><span class="o">,</span> <span class="bp">[]</span><span class="o">,</span> <span class="bp">false</span><span class="o">)</span>
<span class="o">|</span> <span class="bp">false</span> <span class="o">-></span>
<span class="nn">Lwt</span><span class="p">.</span><span class="n">return</span>
<span class="o">(</span> <span class="nn">Fees</span><span class="p">.</span><span class="n">origination_burn</span> <span class="n">ctxt</span> <span class="o">>|?</span> <span class="k">fun</span> <span class="o">(</span><span class="n">ctxt</span><span class="o">,</span> <span class="n">origination_burn</span><span class="o">)</span> <span class="o">-></span>
<span class="o">(</span> <span class="n">ctxt</span><span class="o">,</span>
<span class="o">[</span>
<span class="nn">Receipt</span><span class="p">.</span>
<span class="o">(</span> <span class="nc">Contract</span> <span class="n">payer</span><span class="o">,</span>
<span class="nc">Debited</span> <span class="n">origination_burn</span><span class="o">,</span>
<span class="nc">Block_application</span> <span class="o">);</span>
<span class="o">],</span>
<span class="bp">true</span> <span class="o">)</span> <span class="o">)))</span>
<span class="o">>>=?</span> <span class="k">fun</span> <span class="o">(</span><span class="n">ctxt</span><span class="o">,</span> <span class="n">maybe_burn_balance_update</span><span class="o">,</span> <span class="n">allocated_receiver_contract</span><span class="o">)</span> <span class="o">-></span>
<span class="nn">Contract</span><span class="p">.</span><span class="n">credit</span> <span class="n">ctxt</span> <span class="n">receiver</span> <span class="n">amount</span> <span class="o">>>=?</span> <span class="k">fun</span> <span class="n">ctxt</span> <span class="o">-></span>
<span class="o">...</span>
</code></pre></div>
<p>This implementation makes it difficult to ensure that the tokens have been accurately transferred from the sender to the receiver.
Indeed, to ensure this, it is also necessary to be certain that the function calls between the transfer steps do not modify the balance of the accounts concerned.
Moreover, the <code>balance updates</code> corresponding to debiting the sender and crediting the receiver are done much later after many other instructions.
Furthermore, between debiting the sender and crediting the receiver, a <code>balance update</code> is created to reflect the fact that tokens have been burned to pay the origination cost.
However, the call to <code>Fees.origination_burn</code> does not involve tokens and only involves the accounting of the consumed storage space.
Here the <code>balance update</code> is created in anticipation of tokens being burned at some other location in the code when the following instruction is executed: <code>Fees.burn_storage_fees ctxt ~storage_limit ~payer:...</code>.
Consequently it is more difficult to verify that all balance updates found in block metadata faithfully reflect token movements that have occurred.</p>
<h2 id="token-management-in-ithaca"><a class="toclink" href="#token-management-in-ithaca">Token management in Ithaca</a></h2>
<p>In protocol <strong>Ithaca</strong> token management explicitly consists of three possible operations:</p>
<ul>
<li>Mint tokens and deposit them into an account</li>
<li>Transfer tokens from one account to another</li>
<li>Withdraw tokens from an account and burn them</li>
</ul>
<p>A new module named <a href="https://tezos.gitlab.io/api/odoc/_html/tezos-protocol-alpha/Tezos_raw_protocol_alpha/Token/index.html"><code>Token</code></a> provides functions to perform these operations, and to obtain the balance of an account.
Here, the term account must be understood in a broader sense than in implicit or originated accounts.
For instance, to freeze the deposits of a delegate, deposited funds are withdrawn from the delegate’s implicit account and its balance of frozen tokens is increased.
To make that type of token management operation fit the pattern of a transfer from one account to another, the frozen deposits of a delegate must be viewed as a kind of account from the point of view of the <code>Token</code> module.
Hence, the notion of account needs to be generalized.</p>
<h3 id="a-lightweight-generalization-of-accounts"><a class="toclink" href="#a-lightweight-generalization-of-accounts">A lightweight generalization of accounts</a></h3>
<p>We want to make the notion of a transfer from one account to another more explicit, while minimizing the amount of rewritten code, and hence reducing the risk of introducing bugs.
Hence, the notion of account is generalized in a lightweight fashion.
There are three kinds of accounts: source accounts, container accounts, and sink accounts.</p>
<p><strong>Source accounts</strong> are debited whenever new tokens are minted, and designate fictitious accounts with a virtually infinite balance from which tokens can be withdrawn.
For example, <code>`Nonce_revelation_rewards</code> is the source account of tokens minted to reward delegates for revealing their nonces, and <code>`Liquidity_baking_subsidies</code> is the source account of tokens minted to subsidize the liquidity baking <span class="caps">CPMM</span> contract.</p>
<p><strong>Container accounts</strong> are regular (user and smart contract) accounts, or convenience accounts that hold tokens temporarily (e.g. when parts of a delegate’s funds are frozen).
These accounts have a finite capacity (of a fixed-size integer) and a balance that is increased or decreased whenever they are credited or debited.
The function <code>Token.balance</code> allows to read the balance of a container account.
For example, the account <code>(`Contract c)</code> represents an implicit or originated account <code>c</code>, and the account <code>(`Frozen_deposits d)</code> represents the account of the frozen deposits of the delegate <code>d</code>.</p>
<p><strong>Sink accounts</strong> are credited whenever tokens are burned, and designate fictitious accounts virtually able to receive an unlimited number of tokens.
For example, the sink account <code>`Storage_fees</code> is the receiver of storage fees burned for consuming storage space on the chain, and the sink account <code>`Double_signing_punishments</code> is the receiver of tokens burned as punishment for a delegate that has double baked or double endorsed.</p>
<p>Tokens can be transferred from a sender account (i.e. a source or container account) to a receiver account (i.e. a container or sink account).
The type <code>Token.container</code> represents container accounts.
The type <code>Token.source</code> represents accounts that can play the role of the sender in a transfer of tokens, and similarly, the type <code>Token.sink</code> represents accounts that can play the role of the receiver.
Both <code>Token.source</code> and <code>Token.sink</code> contain <code>Token.container</code> since container accounts can be both sender and receiver of tokens.
Those three types are wrappers that allow the <code>Token</code> module to dynamically dispatch the operations of crediting and debiting corresponding accounts to the right piece of code able to handle the operations.</p>
<h3 id="transferring-tokens"><a class="toclink" href="#transferring-tokens">Transferring tokens</a></h3>
<p>The operations of token management can be performed by invoking the <code>transfer</code> or <code>transfer_n</code> functions of the <code>Token</code> module. A transfer of a given amount from a sender to a receiver simply consists in withdrawing that amount from the sender’s account and crediting the same amount to the receiver’s account. Hence:</p>
<ul>
<li>to mint an amount of tokens and deposit it into a receiver account, one transfers that amount from a source account to the receiver account</li>
<li>to move tokens from one container account to another, one performs a transfer from the former (sender) account to the latter (receiver) account, and</li>
<li>to burn an amount of tokens withdrawn from a given account, one performs a transfer from the sender account to a sink account.</li>
</ul>
<p>Consider the following examples:</p>
<ul>
<li><code>Token.transfer ctxt (`Contract sender) (`Contract receiver) amount</code></li>
<li><code>Token.transfer ctxt `Endorsing_rewards (`Contract d) rewards</code></li>
<li><code>Token.transfer ctxt (`Frozen_deposits d) `Double_signing_punishments amount_to_burn</code>.</li>
</ul>
<p>The first example is the instruction invoked during a transaction operation from a sender contract to a receiver contract.
Here the constructor <code>`Contract</code> allows to construct container accounts that can designate sender accounts as well as a receiver accounts.
The implementation of the transfer functions is such that, by construction, transfers between container accounts leave the total amount of tokens in circulation unchanged.
Also, those functions are the only locations where that property needs to hold.</p>
<p>The second and third examples correspond to the instructions invoked, respectively, to distribute endorsing rewards to a delegate <code>d</code>, and to punish a delegate <code>d</code> for double signing.
Here <code>Endorsing_rewards</code> is a source account that cannot play the role of the receiver in a transfer, and <code>Double_signing_punishments</code> is a sink account that cannot play the role of the sender in a transfer.
Using distinct types of accounts for different types of token transfers makes intent more explicit, more tractable and easier to verify.
Tokens withdrawn from a source account are by definition minted, and tokens sent to sink accounts are by definition burned.
When sticking to this transfer pattern, we can be sure that these are the only ways to respectively increase or decrease the amount of tokens in circulation.
Since all token management operations now follow the <em>transfer pattern</em>, it is easy to list all locations in the protocol where token movements are involved just by running:</p>
<p><code>grep -R "Token.transfer" src/proto_alpha</code>.</p>
<h3 id="balance-updates"><a class="toclink" href="#balance-updates">Balance updates</a></h3>
<p>Whenever tokens are minted, moved, or burned, one or more accounts are debited, and another account is credited.
This results into a sequence of <code>balance updates</code> reporting the amount of tokens debited from and credited to each of the accounts involved.
In Ithaca, balance updates are exclusively generated by the transfer functions of the <code>Token</code> module.
Therefore, by construction, they reflect exactly the token movements that have occurred.
And it is now easier to assess that they are exhaustive and correct just by reviewing the implementation of the <code>Token</code> module.</p>
<p>In block metadata, the field <code>balance_updates</code> contains balance updates generated by the invocations of token transfer functions.
Each invocation of those functions generates a list of balance updates starting with a series of debits, and ending with a credit matching those debits.
Typically, this field contains a flat list resulting from the concatenation of one or more lists of balance updates.
Consider for example the following flat list of balance updates:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="nt">"kind"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="p">,</span><span class="w"> </span><span class="err">...</span><span class="p">,</span><span class="w"> </span><span class="nt">"change"</span><span class="p">:</span><span class="w"> </span><span class="s2">"-100"</span><span class="p">,</span><span class="w"> </span><span class="nt">"origin"</span><span class="p">:</span><span class="w"> </span><span class="s2">"block"</span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="nt">"kind"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="p">,</span><span class="w"> </span><span class="err">...</span><span class="p">,</span><span class="w"> </span><span class="nt">"change"</span><span class="p">:</span><span class="w"> </span><span class="s2">"100"</span><span class="p">,</span><span class="w"> </span><span class="nt">"origin"</span><span class="p">:</span><span class="w"> </span><span class="s2">"block"</span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="nt">"kind"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="p">,</span><span class="w"> </span><span class="err">...</span><span class="p">,</span><span class="w"> </span><span class="nt">"change"</span><span class="p">:</span><span class="w"> </span><span class="s2">"-125"</span><span class="p">,</span><span class="w"> </span><span class="nt">"origin"</span><span class="p">:</span><span class="w"> </span><span class="s2">"block"</span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="nt">"kind"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="p">,</span><span class="w"> </span><span class="err">...</span><span class="p">,</span><span class="w"> </span><span class="nt">"change"</span><span class="p">:</span><span class="w"> </span><span class="s2">"-75"</span><span class="p">,</span><span class="w"> </span><span class="nt">"origin"</span><span class="p">:</span><span class="w"> </span><span class="s2">"block"</span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="nt">"kind"</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="p">,</span><span class="w"> </span><span class="err">...</span><span class="p">,</span><span class="w"> </span><span class="nt">"change"</span><span class="p">:</span><span class="w"> </span><span class="s2">"200"</span><span class="p">,</span><span class="w"> </span><span class="nt">"origin"</span><span class="p">:</span><span class="w"> </span><span class="s2">"block"</span><span class="p">}</span><span class="w"> </span><span class="p">]</span>
</code></pre></div>
<p>This list reports that two transfers have occurred: <code>100</code> mutez are transferred from one account to another, and a total of <code>200</code> mutez are transferred from two accounts to a third.
For a more complete description of the format for balance updates, see this <a href="https://tezos.gitlab.io/012/token_management.html">page</a> of the documentation.</p>
<h2 id="conclusions-future-work"><a class="toclink" href="#conclusions-future-work">Conclusions <span class="amp">&</span> Future Work</a></h2>
<p>In <strong>Ithaca</strong> we have centralized the management of native protocol token movements in a dedicated module so that token transfers are correct by construction.
Basically we have implemented some of the ideas mentioned <a href="https://forum.tezosagora.org/t/modularizing-accounting-in-the-protocol/1440">here</a> and <a href="https://gitlab.com/tezos/tezos/-/issues/316">here</a> in a lightweight manner so as to minimize the risk of introducing bugs.
The <code>transfer</code> pattern is now the only uniform means of minting, moving, or burning tokens in the Tezos protocol.
By construction, balance updates in block metadata faithfully and exhaustively reflect all token transfers that have occurred when the block has been applied.
To test this refactoring work, we have very much relied on our existing suite of unit and integration tests, and we have also written more unit tests to check the expected properties of the token transfer functions.</p>
<p>In some cases quite sensitive changes were necessary before we could apply the new transfer pattern to manage tokens, while preserving previous behavior.
For instance, this was the case for the burning storage fees. Here it was important to preserve the validity of transactions where the sender’s balance is sufficient to pay the fees only when all internal transactions have been processed.
To test such changes, we have back-ported them to previous protocols, and the chain has been replayed to ensure that the previous behavior has been preserved.</p>
<p>Our next work on the subject of token management will be to extend the <code>Token</code> module so that it also manages token delegations.
The staking balance of a delegate can only be changed when token owners delegate or transfer their tokens, but updating staking balances is currently performed at various locations after each token transfer that affects staking balances.
With the refactoring described above it will be possible to update the staking balance once and for all in the transfer functions of the <code>Token</code> module.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:native">
<p>Tez are the Tezos blockchain’s <em>native protocol</em> tokens. We use <em>native protocol</em> here to make explicit that in this article we focus on the former and not on, e.g., <em>digital assets</em> implemented using Tezos smart contacts following the <span class="caps">FA1</span>.2 and <span class="caps">FA2</span> token standards. <a class="footnote-backref" href="#fnref:native" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Announcing “Ithaca 2”2022-01-19T17:00:00+01:002022-01-19T17:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2022-01-19:/announcing-ithaca-2.html<p><em>This is a joint post from TriliTech, Nomadic Labs, Marigold, Oxhead Alpha, Tarides, DaiLambda, Functori <span class="amp">&</span> Tweag.</em></p>
<p>We are happy to announce <a href="http://tezos.gitlab.io/protocols/012_ithaca.html">Ithaca 2</a>, a revised version of Ithaca that contains an important performance improvement of metadata RPCs.</p>
<p>(As is usual, Ithaca 2’s “true name” is its hash, which is <code>Psithaca2MLRFYargivpo7YvUr7wUDqyxrdhC5CQq78mRvimz6A</code>.)</p>
<p>Ithaca 2’s sole difference with Ithaca is the <strong>deprecation of a redundant field in an error message</strong>. When the interpretation of the Michelson script of a smart contract results in a runtime error, the script code will not be passed anymore to the error message. This will save a lot of space because the script is typically large and is duplicated in each error message - i.e. each time the smart contract will be called and its execution results in a runtime error. Furthermore, the script code can already be fetched from the contract address, so there is no need to record this information again. Storing script code in the block metadata is an <strong>anomalous behavior</strong> and the deprecation of this redundant field will result in an <strong>important reduction of disk usage</strong>.</p>
<p>Other features of Ithaca 2 are contained in Ithaca and we refer to our <a href="https://research-development.nomadic-labs.com/announcing-tezos-9th-protocol-upgrade-proposal-ithaca.html">previous announcement</a>.</p>
<p>We also encourage you to look at the <a href="https://tezos.gitlab.io/protocols/012_ithaca.html">changelog</a> for a full description of the contents of the proposal.</p>
<hr>
<p>As it was the case with Ithaca, testing is critical. A testnet for the Ithaca 2 protocol will launch in the coming days. It is critical to have as many bakers as possible participating in this testnet, by running nodes, producing blocks and deploying apps. We are looking for more bootstrap bakers to participate from day one. If you are interested, please <a href="https://tezos-kiln.org/joinbakingslack">join the baking slack</a> and make yourselves known in the <code>#test-networks</code> channel.</p>
<p>Once more, we strongly encourage you to test your own Tezos-based applications to check for compatibility problems. Ithaca 2, and the configuration for its test network, will be included in the next release candidate of Octez version 12.</p>
<p>Similarly to Ithaca, should the Ithaca 2 protocol proposal be accepted by the community, the following minimal version of Tezos node (shell) software will be necessary to participate in the consensus, due to necessary changes introduced to the protocol environment: v12 of Octez, or v2 of TezEdge.</p>Tenderbake has been injected2021-12-21T12:00:00+01:002021-12-21T12:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2021-12-21:/tenderbake-has-been-injected.html<p>Outline of what the Tenderbake upgrade means for you, and what benefits it will bring</p><p>You may recall from earlier this year our <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">in-depth discussion of Tenderbake</a>, our new classic-style consensus algorithm that offers quick and deterministic finality and safety under asynchrony.</p>
<p>Since then, and in collaboration with <a href="https://functori.com/">Functori</a>, we have prepared Tenderbake for inclusion in the next Tezos protocol upgrade proposal <a href="https://research-development.nomadic-labs.com/announcing-tezos-9th-protocol-upgrade-proposal-ithaca.html">Ithaca</a>.
A big thank you to the incredible and amazing Functori for their incredibly amazing collaboration!</p>
<p>In this post, we will complement <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">our previous discussion of Tenderbake</a> with a recap of some practical aspects of the new consensus algorithm: what does Tenderbake mean for you, how can you use it, and what benefits will it bring?
We will overview the most important points here, and you can find further details in</p>
<ul>
<li><a href="https://tezos.gitlab.io/012/consensus.html">the full Tenderbake documentation</a>, and</li>
<li>the <a href="https://tezos.gitlab.io/protocols/tenderbake.html">Tenderbake changelog</a>.</li>
</ul>
<h2 id="overview"><a class="toclink" href="#overview">Overview</a></h2>
<p>Recall that when determining which block to add to the blockchain, Tenderbake proceeds in <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html#levels-are-composed-of-rounds">rounds</a>, starting from round 0:</p>
<ul>
<li>If there is enough timely participation at round 0, then consensus is reached and a decision is made.</li>
<li>If there is not enough timely participation<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup> at round 0 then we go to round 1, and so forth.</li>
</ul>
<p>Provided that at least two thirds of the total active stake participates honestly in consensus, then a decision is eventually taken.<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup>
In the current implementation of Tenderbake the duration of each round increments by 15 seconds, starting from 30 seconds: thus the deadline for participation in round 0 is 30 seconds, that for round 1 is 45 seconds after that, and so on.
So in normal conditions, when consensus is reached promptly at round 0 every time, we can expect Tenderbake to add one block every 30 seconds.</p>
<p>Note that:</p>
<ul>
<li>Tenderbake has deterministic finality <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html#deterministic-finality-bft-style">after just two blocks</a>. In normal conditions, when the network is healthy, decisions are made at round 0, after 30 seconds. This means that <strong>in normal conditions the time to finality is about one minute</strong>.</li>
<li>Finality is <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html#tenderbake-has-quick-deterministic-finality">guaranteed under an assumption that at most one third of the active stake is Byzantine</a> (a Byzantine participant is one that is either offline and so unresponsive, or responsive but not following the rules). This is a standard assumption, but it does have the corollary that <strong>the chain will halt if more than one third of the active stake is offline.</strong><sup id="fnref:off"><a class="footnote-ref" href="#fn:off">3</a></sup></li>
</ul>
<p>Speaking of bakers — baker daemons now endorse as well.
Indeed, Tenderbake bids farewell to the concept of endorser daemon from the Emmy family of consensus algorithms which Tezos has used to date.
Goodbye endorser daemon; thank you for the years of service!</p>
<p>The baker also emits a new kind of consensus operation called a <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html#why-do-we-need-preendorsements">preendorsement</a>.</p>
<p>Aside from the new kind of consensus operations, there are changes to the block metadata and RPCs, so Tezos tool developers — for example, developers of block explorers and indexers — are invited to check the <a href="https://tezos.gitlab.io/protocols/tenderbake.html#metadata">metadata and <span class="caps">RPC</span> changes</a> so that their tools can be seamlessly updated.</p>
<p>Finally, there are notable improvements to the incentives mechanism, which we describe next:</p>
<h2 id="the-incentives-mechanism"><a class="toclink" href="#the-incentives-mechanism">The incentives mechanism</a></h2>
<h3 id="the-deposit-scheme"><a class="toclink" href="#the-deposit-scheme">The deposit scheme</a></h3>
<h4 id="in-emmy"><a class="toclink" href="#in-emmy">In Emmy*</a></h4>
<p>In <a href="https://tezos.gitlab.io/011/consensus.html#economic-incentives">Emmy*</a>, each time a delegate wants to bake or endorse, it must put down a deposit as a guarantee of honest behaviour.
That is: in Emmy* there is <a href="https://tezos.gitlab.io/011/consensus.html#slashing">a deposit for each individual baking and endorsement action</a>, which is computed such that security deposits are around 10% of the total stake.<sup id="fnref:ref"><a class="footnote-ref" href="#fn:ref">4</a></sup>
Double signing in Emmy* is penalised by forfeiting the entirety of the deposits made by the delegate during the cycle when the double signing was made.<sup id="fnref:cycle"><a class="footnote-ref" href="#fn:cycle">5</a></sup></p>
<p>This design has some drawbacks:</p>
<ul>
<li>If a delegate becomes over-delegated (meaning that their stake is not sufficient for their required deposits) then this may lead to the delegate <a href="https://tezos.gitlab.io/introduction/howtorun.html#over-delegation">missing slots and thus slowing down the chain</a>.</li>
<li>Handling deposits requires I/O operations. This imposes communication and computational overheads, which may slow down block validation.<sup id="fnref:nodes"><a class="footnote-ref" href="#fn:nodes">6</a></sup></li>
<li>Penalties for double signing (see the discussion of <a href="#slashing">slashing</a> below) are not proportional to the double signer’s stake.</li>
</ul>
<h4 id="in-tenderbake"><a class="toclink" href="#in-tenderbake">In Tenderbake</a></h4>
<p>Tenderbake takes steps to address the issues above.</p>
<p>In Tenderbake’s <a href="https://tezos.gitlab.io/012/consensus.html#validator-selection-staking-balance-active-stake-and-frozen-deposits">new deposits scheme</a>, delegates still need to put down deposits as a guarantee against <a href="#slashing">double signings</a>, but this deposit is now based on the delegate’s stake rather than on individual baking/endorsement actions.
Deposits are frozen when baking and (pre)endorsement rights are allocated to stakeholders, five cycles in advance.</p>
<p>The deposit amount is determined by the stake a delegate has during the so-called <em>active</em> cycles: two cycles in the past and five cycles in the future.
To be precise, a delegate’s frozen deposit at cycle <code>c</code> is 10% of the highest stake during cycles <code>c-2</code> to <code>c+5</code>.</p>
<p>Delegates can put an upper limit on their deposit by using the client command <a href="https://tezos.gitlab.io/012/consensus.html#validator-selection-staking-balance-active-stake-and-frozen-deposits"><code>set deposit limit</code></a>. In particular a delegate can set this limit to zero, and then the delegate will lose its baking rights in 7 cycles.
To be able to bake again, the delegate would have to reset the deposits limit or simply unset it (with the client command <code>unset deposit limit</code>).</p>
<h3 id="slashing"><a class="toclink" href="#slashing">Slashing</a></h3>
<p>In Tenderbake, we say a delegate <strong>double signs</strong> when it bakes or (pre)endorses <em>two</em> blocks at the <em>same level and round</em>, but with <em>different</em> payload hashes.<sup id="fnref:payload"><a class="footnote-ref" href="#fn:payload">7</a></sup>
Double signing is penalised by <strong>slashing</strong> (forfeiting) a part of the delegate’s deposit:</p>
<ul>
<li>Double baking is penalised by slashing 640 ꜩ (or the whole deposit, if less than 640 ꜩ).</li>
<li>Double (pre)endorsement is penalised by slashing half of the deposit. This means that a wrongdoer who double (pre)endorses twice during a single cycle will lose their entire deposit, and thus lose baking rights for the rest of the cycle.</li>
</ul>
<p>These penalties reflect the relative severities of double baking versus double (pre)endorsement: double baking must be penalised because it wastes resources, but double (pre)endorsement could fork the chain, which would be much more serious.<sup id="fnref:less-serious"><a class="footnote-ref" href="#fn:less-serious">8</a></sup></p>
<h3 id="rewards"><a class="toclink" href="#rewards">Rewards</a></h3>
<p>In Tenderbake, just as in <a href="https://tezos.gitlab.io/011/consensus.html#economic-incentives">Emmy*</a>:</p>
<ul>
<li>Participation in consensus (both baking and endorsement) is rewarded.</li>
<li>A delegate must have a minimal stake to participate.<sup id="fnref:3"><a class="footnote-ref" href="#fn:3">9</a></sup> (The minimal stake for becoming a baker is 8000 ꜩ in Emmy* and 6000 ꜩ in Tenderbake.)</li>
</ul>
<p>However in Tenderbake, baking rewards and fees can be dispensed immediately, thanks to <a href="#the-deposit-scheme">the new deposit scheme</a>.</p>
<h3 id="the-block-its-payload-and-baking-rewards"><a class="toclink" href="#the-block-its-payload-and-baking-rewards">The block, its payload, and baking rewards</a></h3>
<p>In Tenderbake we distinguish between the <em>payload producer</em> and the <em>block producer</em>:</p>
<ul>
<li>The payload producer is the baker who selects the non-consensus operations to be included in the block; we refer to these operations as the <em>block’s payload</em>.</li>
<li>The block producer is the baker who signs the block.</li>
</ul>
<p>The payload producer is typically the same as the block producer — but not necessarily!
For instance, if consensus is not reached within a given round and the baker has committed to the block proposed at the given round by endorsing the proposed block, then it might be that at a later round the baker is forced to propose a block with the same payload.
In such a case <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html#first-round-vs-subsequent-rounds">the payload producer may differ from the block producer</a>.</p>
<p>Given this separation:</p>
<ul>
<li>Operation fees as well as a fixed baking reward of 10 ꜩ go to the payload producer, because the payload producer selected the operations.</li>
<li>A bonus for including endorsements above the required threshold goes to the block producer, to incentivize him/her to play fairly.</li>
</ul>
<p>To be precise, a block producer who includes endorsements corresponding to <code>x</code> extra endorsement slots receives a bonus of <code>0.004286 * max (0, x - 4667)</code> ꜩ, where</p>
<ul>
<li><code>4667 = 2/3 * 7000 + 1</code> represents the required threshold for agreement,<sup id="fnref:23"><a class="footnote-ref" href="#fn:23">10</a></sup> and</li>
<li><code>7000</code> is the value fixed for the total number of endorsement slots (see parameters <code>consensus_committee_size</code> and <code>consensus_threshold</code> in <a href="https://tezos.gitlab.io/alpha/consensus.html#consensus-related-protocol-parameters">the documentation</a>).</li>
</ul>
<h3 id="endorsement-rewards"><a class="toclink" href="#endorsement-rewards">Endorsement rewards</a></h3>
<p>Endorsement rewards are distributed at the end of each cycle, because only endorsers actively participating during the given cycle are rewarded.
Such actively participating endorsers are known as <em>present</em><sup id="fnref:present"><a class="footnote-ref" href="#fn:present">11</a></sup> endorsers.</p>
<p>An endorser is considered <strong>present</strong><sup id="fnref:4"><a class="footnote-ref" href="#fn:4">12</a></sup> when the number of slots corresponding to its endorsements which are included in blocks is at least two thirds of its total slots during the cycle.
Therefore an endorser may be rewarded even if some of its endorsements are not included.</p>
<p>A present endorser is rewarded proportionally to its stake, and not proportionally to its allocated slots.
To be precise, a present endorser with <code>x</code> percent of the total active stake during a cycle, is rewarded with <code>0.002857 * x * 8192 * 7000</code> ꜩ where</p>
<ul>
<li><code>8192</code> is the number of blocks per a cycle and</li>
<li><code>7000</code> is the <code>consensus_committee_size</code> mentioned above.</li>
</ul>
<h2 id="happy-tenderbaking-remember-to-update"><a class="toclink" href="#happy-tenderbaking-remember-to-update">Happy Tenderbaking (remember to update!)</a></h2>
<p>Please remember to update to the latest version of <a href="https://tezos.gitlab.io/releases/version-12.html">Octez</a> — otherwise, your baker might end up being a Byzantine baker simply by virtue of running out-of-date software, and nobody wants that!
You can learn more about Tenderbake in <a href="https://tezos.gitlab.io/012/consensus.html">the full documentation</a> and <a href="https://tezos.gitlab.io/protocols/tenderbake.html">changelog</a>.</p>
<p>That’s it!
Stay tuned for another blog post on testing Tenderbake.
Until then, we wish you all the best for the New Year: may all your chains reach prompt and productive consensus!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>Perhaps the network is running slow … <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>… for instance, once the network outage passes. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:off">
<p>An offline baker is (as standard) considered to be a Byzantine baker — a <em>Byzantine baker</em> is one that, by definition, does not properly act according to the consensus algorithm, and the simplest way to not act according to the consensus algorithm is to not act at all! Whether this is by accident (poor network) or design (actively hostile to the network) is, from the point of view of designing a consensus algorithm, unimportant. <a class="footnote-backref" href="#fnref:off" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:ref">
<p>See <a href="https://tezos.gitlab.io/active/consensus.html#consensus-protocol-parameters">this discussion of the consensus parameters</a>: <em>“Since deposits are locked for a period of PRESERVED_CYCLES, one can compute that, at any given time, about ((BLOCK_SECURITY_DEPOSIT + ENDORSEMENT_SECURITY_DEPOSIT * ENDORSERS_PER_BLOCK) * (PRESERVED_CYCLES + 1) * BLOCKS_PER_CYCLE) tokens of all staked tokens should be held as security deposits. For instance, if the amount of staked tokens is 720,000,000 ꜩ, then roughly 8.74% of this amount is stored in security deposits.”</em> <a class="footnote-backref" href="#fnref:ref" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:cycle">
<p>A <a href="https://tezos.gitlab.io/012/glossary.html#cycle">cycle</a> in Tezos is a certain number of blocks. This is <a href="https://tezos.gitlab.io/active/proof_of_stake.html#proof-of-stake-parameters">8192</a> blocks, and will stay the same in Tenderbake. At a typical block time of 30 seconds, this comes to about three days. <a class="footnote-backref" href="#fnref:cycle" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:nodes">
<p>The bottleneck here is <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html#the-tezos-architecture">the node</a>. Each baking/endorsement event requires its own small deposit, which imposes its own small overhead. A node that is busy processing the I/O operations needed to update the data structure storing the security deposits, is a node with less time for validating its next block. <a class="footnote-backref" href="#fnref:nodes" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:payload">
<p>A block’s <em>payload</em> is the sequence of non-consensus operations contained in the block. Preendorsements and endorsements contain the hash of the payload they refer to. <a class="footnote-backref" href="#fnref:payload" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn:less-serious">
<p>Even if there are two blocks at the same level and round, honest bakers will (pre)endorse only one of them so safety is preserved. <a class="footnote-backref" href="#fnref:less-serious" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>The reader might be familiar with the notion of <code>token_per_roll</code> in Emmy* which was used to denote the minimal stake needed to participate in consensus. In Emmy*, <a href="https://tezos.gitlab.io/active/proof_of_stake.html#delegates-rights-selection">rolls</a> played a role in the computation of baking and endorsement rights.
In Tenderbake, this is no longer the case because rights are computed from delegates’ stake. Thus <a href="https://tezos.gitlab.io/protocols/tenderbake.html#protocol">rolls are only used to determine the voting power of a delegate</a>. <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 9 in the text">↩</a></p>
</li>
<li id="fn:23">
<p>A block at level <code>l+1</code> must include endorsements for a block at level <code>l</code> as a proof that <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html#first-round-vs-subsequent-rounds">agreement has been reached at level <code>l</code></a>.
These endorsements must correspond to at least two thirds of the total slots. <a class="footnote-backref" href="#fnref:23" title="Jump back to footnote 10 in the text">↩</a></p>
</li>
<li id="fn:present">
<p>Note that this notion is different from the notion of <a href="https://tezos.gitlab.io/012/proof_of_stake.html#active-and-passive-delegates">active delegates</a>. <a class="footnote-backref" href="#fnref:present" title="Jump back to footnote 11 in the text">↩</a></p>
</li>
<li id="fn:4">
<p>To monitor participation, the protocol provides a new <span class="caps">RPC</span>, unsurprisingly dubbed <a href="https://tezos.gitlab.io/012/rpc.html#get-block-id-context-delegates-pkh-participation">participation</a>. <a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 12 in the text">↩</a></p>
</li>
</ol>
</div>Announcing Tezos’ 9th protocol upgrade proposal “Ithaca”2021-12-20T15:00:00+01:002021-12-20T15:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2021-12-20:/announcing-tezos-9th-protocol-upgrade-proposal-ithaca.html<p><em>This is a joint post from TriliTech, Nomadic Labs, Marigold, Oxhead Alpha, Tarides, DaiLambda, Functori <span class="amp">&</span> Tweag.</em></p>
<p>We were proud to see Hangzhou go live on chain on December 4th, 2021.
In keeping with our policy of proposing upgrades on a regularly scheduled basis, we’re happy to announce our latest Tezos protocol proposal, Ithaca.</p>
<p>(As is usual, Ithaca’s “true name” is its hash, which is <code>PsiThaCaT47Zboaw71QWScM8sXeMM7bbQFncK9FLqYc6EKdpjVP</code>).</p>
<p>Ithaca contains two major updates to the protocol, as well as numerous minor improvements.
Below we discuss some of the most interesting and important changes.</p>
<h2 id="tenderbake"><a class="toclink" href="#tenderbake">Tenderbake</a></h2>
<p><a href="http://tezos.gitlab.io/012/consensus.html">Tenderbake</a> is a major update to the Tezos consensus algorithm.
Like Tendermint, Tenderbake brings fast deterministic finality to the Tezos protocol.</p>
<p>Tenderbake comes with a set of important changes:</p>
<ol>
<li>
<p>The protocol moves away from a roll-based model to an optimized stake-based model to allocate rewards: bakers will receive rewards depending on their current stake instead of the number of rolls they own.</p>
</li>
<li>
<p>A reduction in the minimal number of tokens required to be selected as a validator would be implemented: from 8,000 tez to 6,000 tez.
This minimal stake of 6,000 tez remains necessary for performance reasons.</p>
</li>
<li>
<p>The baking and endorsement rewards mechanism has been reworked (c.f. <a href="http://tezos.gitlab.io/012/consensus.html#rewards">rewards documentation</a>).
In particular, baking rewards will be credited instantaneously, and not frozen for 5 cycles as is the case with Emmy*.
Furthermore, there will no longer be a variance for endorsement rewards.
The total sum of endorsement rewards for a cycle will be fully distributed at the end of the same cycle, <strong>provided</strong> delegates have at least 2/3 of their endorsement slots included in blocks.</p>
</li>
<li>
<p>A new security deposit mechanism is introduced: delegates are required to <em>freeze</em>, at minimum, 10% of their stake in advance in order to obtain baking and endorsement rights.
A new operation <code>Set_deposit_limit</code> is also introduced to manually manage this limit.</p>
</li>
<li>
<p>The number of endorsement slots per block has been bumped from 256 to 7,000: this means that a delegate with the minimum amount of tokens will participate every 10 blocks on average.
The node’s storage layer and prevalidator have been optimized to handle the charge, with the <em>precheck</em> feature also contributing to the increase in performance.
The number of endorsement operations, which will continue to endorse multiple slots, will be proportional to the number of validators in the network, i.e. around 500.</p>
</li>
<li>
<p>Since Tenderbake is modeled after classical <span class="caps">BFT</span> consensus algorithms, it favors safety over liveness and requires active participation of validators holding 2/3 of the stake in order for the chain to progress.</p>
</li>
</ol>
<p>This consensus algorithm also offers the possibility to easily reduce the minimal time between blocks, which may be proposed in future Tezos protocol amendments.</p>
<h2 id="precheck-of-operations"><a class="toclink" href="#precheck-of-operations">Precheck of operations</a></h2>
<p>The new version of the protocol will enable the prechecking of operations.
This is not a feature of the Ithaca protocol proposal <em>per se</em>, but it rather consists of a new set of functions which are exposed by the economic protocol, and which can be used by any Tezos <em>shell</em> (e.g., Octez and TezEdge) to avoid fully executing manager operations before gossiping them through the network.</p>
<p>The feature serves mainly one purpose: increasing the number of operations gossiped over the Tezos network.
It is a prequel to further optimizations that should increase the transaction throughput over the Tezos network.</p>
<h2 id="liquidity-baking"><a class="toclink" href="#liquidity-baking">Liquidity Baking</a></h2>
<p>Ithaca includes an increase to the liquidity baking sunset level of 819,200
blocks, or twenty voting periods, roughly an additional ten months.
This bigger increase will avoid needing to worry about the sunset level for the next few protocol amendments.
Also, to balance this increase, the threshold for activating the escape hatch is lowered from 50% to 33%.</p>
<hr>
<p>We invite you to look at the <a href="https://tezos.gitlab.io/protocols/012_ithaca.html">changelog</a> for a full description of the contents of the proposal.</p>
<p>Ithaca is the biggest update to Tezos to date, and testing is critical. A testnet for Ithaca protocol named Ithacanet will launch in the coming days.
It is critical to have as many bakers as possible participating in this testnet, by running nodes, producing blocks and deploying apps.
We are looking for more bootstrap bakers to participate from day one.
If you are interested, please <a href="https://tezos-kiln.org/joinbakingslack">join the baking slack</a> and make yourselves known in the test-networks channel.</p>
<p>Furthermore, we strongly encourage you to test your own Tezos-based applications to check for compatibility problems with Ithaca.
Ithaca, and the configuration for its test network Ithacanet, will be included in the version 12 of Octez.</p>
<p>Should the Ithaca protocol proposal be accepted by the community, the following minimal version of Tezos node (<em>shell</em>) software will be necessary to participate in the consensus, due to necessary changes introduced to the protocol environment: v12 of Octez, or v2 of TezEdge.</p>
<p>If Ithaca is adopted, the next proposal (which likely will have a name starting with the letter “J”) should be proposed and enter the Tezos amendment process next year.</p>
<p>Over the course of the coming months, our teams also intend to continue to develop and propose amendments to increase performance, lower gas consumption, reduce block times, and increase the overall Tezos network’s throughput — as measured, for example, in <em>transactions per seconds</em>, or <em>smart contract invocations per second</em>.
We are all excited to continue developing the future of Tezos.</p>Meanwhile at Nomadic Labs #132021-10-28T15:00:00+02:002021-10-28T15:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-10-28:/meanwhile-at-nomadic-labs-13.html<p>A summary of Nomadic Labs activities in Q3 2021</p><p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>Welcome to our <a href="https://research-development.nomadic-labs.com/blog.html?category=meanwhile">meanwhile series</a>, the ongoing story of Nomadic Labs’ amazing adventures in the Tezos blockchain space.
This post is a recap of our activities in the third quarter of 2021, following on from our
<a href="https://research-development.nomadic-labs.com/a-review-of-nomadic-labs-in-2020.html">2020 recap</a> and our <a href="https://research-development.nomadic-labs.com/meanwhile-at-nomadic-labs-12.html">2021 Q2 Meanwhile</a>.
As always, you can find out more about us here:
<a href="https://twitter.com/labosnomades?lang=en">Twitter @LabosNomades</a> ~
<a href="https://www.nomadic-labs.com">Website</a> ~
<a href="https://fr.linkedin.com/company/nomadic-labs">LinkedIn</a> ~
<a href="https://research-development.nomadic-labs.com/blog.html">Technical blog</a> ~
<a href="https://gitlab.com/nomadic-labs">GitLab repo</a>.</p>
<h3 id="table-of-contents-q3-2021"><a class="toclink" href="#table-of-contents-q3-2021">Table of contents (Q3 2021)</a></h3>
<ul>
<li><a href="#octez">Octez</a></li>
<li><a href="#mi-cho-coq">Mi-Cho-Coq</a></li>
<li><a href="#umami">Umami</a></li>
<li><a href="#protocol-upgrade-granada-activated-hangzhou-proposed-and-testnet-launched">Protocol upgrade: Granada activated, Hangzhou proposed and testnet launched</a></li>
<li><a href="#adoption-and-support">Adoption and Support</a></li>
<li><a href="#training">Training</a></li>
<li><a href="#technical-documentation">Technical documentation</a></li>
<li><a href="#phd-student-intern-and-apprentice-interviews">PhD student, Intern, and apprentice interviews</a></li>
<li><a href="#media-interviews-and-academic-papers">Media Interviews and Academic Papers</a></li>
<li><a href="#nl-research-seminars-and-blog-posts"><span class="caps">NL</span> research seminars and blog posts</a></li>
<li><a href="#sponsorship">Sponsorship</a></li>
<li><a href="#contract-calls">Contract calls</a></li>
<li><a href="#a-la-prochaine">À la prochaine</a></li>
</ul>
<h2 id="octez"><a class="toclink" href="#octez">Octez</a></h2>
<p>You may recall from <a href="https://research-development.nomadic-labs.com/meanwhile-at-nomadic-labs-12.html#octez">Meanwhile #12</a> that we announced the christening of <a href="https://research-development.nomadic-labs.com/announcing-octez.html">Octez</a> — the veteran implementation of Tezos which had previously been known just by its version number and by <a href="https://gitlab.com/tezos/tezos/">a GitLab repo</a>.</p>
<p>The Octez team were active in Q3 2021, releasing <a href="https://forum.tezosagora.org/t/releasing-version-9-4-of-octez/3420">Version 9.4 (3 July)</a>, <a href="https://forum.tezosagora.org/t/releasing-version-9-5-of-octez/3508">9.5 (29 July)</a>, <a href="https://forum.tezosagora.org/t/releasing-version-9-6-of-octez/3543">9.6 (6 August)</a>, and <a href="https://forum.tezosagora.org/t/releasing-version-9-7-of-octez/3547">9.7 (7 August)</a>; then <a href="https://forum.tezosagora.org/t/releasing-v10-0-rc3-octez/3558">10.0~rc3 (10 August)</a>, <a href="https://forum.tezosagora.org/t/releasing-version-10-0-of-octez/3585">10.0 (19 August)</a> and then <a href="https://forum.tezosagora.org/t/releasing-version-10-1-of-octez/3616">Version 10.1 (26 August)</a>.
Changelogs are <a href="http://tezos.gitlab.io/CHANGES.html">here</a>.
To top this off, we released the release candidate <a href="http://tezos.gitlab.io/CHANGES.html#version-11-0-rc1">Octez 11.0~rc1</a> on 22 September.</p>
<p>The currently-active version at time of writing (end October) is <a href="https://forum.tezosagora.org/t/releasing-version-10-3-of-octez/3806">Octez 10.3</a>.
<a href="https://gitlab.com/tezos/tezos/">Here is the Octez GitLab repo</a>.
Feel free to <a href="https://tezos.gitlab.io/introduction/howtoget.html">get the Octez Tezos implementation</a> and join the Tezos blockchain!</p>
<h2 id="mi-cho-coq"><a class="toclink" href="#mi-cho-coq">Mi-Cho-Coq</a></h2>
<p>We are proud to announce that <a href="https://research-development.nomadic-labs.com/introducing-mi-cho-coq-v10.html">Mi-Cho-Coq version 1.0</a> was released on 2 July.</p>
<p><a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/releases/version-1.0">Mi-Cho-Coq</a> is a free and open-source library for verifying the correctness of Michelson smart contracts in Coq using weakest-precondition calculus.
It is a Coq library which models all aspects of the Michelson language: its syntax, its type system, and its semantics.<sup id="fnref:deep"><a class="footnote-ref" href="#fn:deep">1</a></sup></p>
<p>For concrete applications of this powerful tool, see for example <a href="https://research-development.nomadic-labs.com/formally-verifying-a-critical-smart-contract.html#verifying-spending-limit-in-mi-cho-coq">verification of a spending-limit contract</a>, of <a href="https://research-development.nomadic-labs.com/fa12-approvable-ledger-formal-verification-by-nomadic-labs.html">the <span class="caps">FA1</span>.2 token standard</a> (see also <a href="https://arxiv.org/abs/2109.09451">an associated paper in <span class="caps">FMBC</span>‘21</a>), and of several versions of the Dexter decentralised exchange (<a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/merge_requests/100">Dexter v2</a> and <a href="https://research-development.nomadic-labs.com/progress-report-on-the-verification-of-liquidity-baking-smart-contracts.html">Liquidity Baking</a>).</p>
<h2 id="umami"><a class="toclink" href="#umami">Umami</a></h2>
<p>The <a href="https://umamiwallet.com">Umami wallet</a> was released in April 2021 and is an all-in-one Tezos <a href="https://en.wikipedia.org/wiki/Cryptocurrency_wallet">cryptocurrency wallet</a> for both beginner and advanced users.
At time of writing the current version is <a href="https://umamiwallet.com/#download">version 0.5.3</a>, available for macOS, Linux, and Windows.</p>
<p>We are pleased to announce that during Q3 2021</p>
<ul>
<li><a href="https://medium.com/airgap-it/umami-wallet-support-beacon-sdk-v2-3-0-f12f3fb5f67f">Beacon <span class="caps">SDK</span> v2.3.0 supports the Umami wallet</a>, and</li>
<li><a href="https://medium.com/umamiwallet/umami-ledger-hardware-590ff0eddd6c">Umami wallet hardware wallet integration</a> was released for <a href="https://www.ledger.com">Ledger devices</a>.</li>
</ul>
<p>Umami was built <a href="https://medium.com/umamiwallet/introducing-the-umami-wallet-4c5d4201ce5">by OCaml developers for OCaml developers</a> using <a href="https://en.wikipedia.org/wiki/Reason_(programming_language)">Reason</a> (formerly ReasonML) and supports all the native features of the Tezos protocol, including multiple accounts, tokens, batch transactions, and delegation — with <a href="https://umamiwallet.com/#features">more features</a> in the pipeline.
For more information see <a href="https://medium.com/umamiwallet">a Umami Wallet page on Medium</a>, a <a href="https://medium.com/umamiwallet/introducing-the-umami-wallet-4c5d4201ce5">short essay on the purpose of Umami</a>, and the <a href="https://gitlab.com/nomadic-labs/umami-wallet/umami">Umami GitLab repo</a>.</p>
<h2 id="protocol-upgrade-granada-activated-hangzhou-proposed-and-testnet-launched"><a class="toclink" href="#protocol-upgrade-granada-activated-hangzhou-proposed-and-testnet-launched">Protocol upgrade: Granada activated, Hangzhou proposed and testnet launched</a></h2>
<p>The Tezos <em>economic protocol</em> enjoys <a href="https://forum.tezosagora.org/t/regular-scheduling-for-our-tezos-proposals/2331">regular upgrades</a>.
How this happens concretely is that a <a href="https://tezos.com/docs/learn/what-is-tezos#self-amendment">self-amendment mechanism</a> is activated to propose an upgrade to the protocol — and because Tezos is an open community, protocol upgrades are approved by community vote.
This means that upgrades can only happen when you, the Tezos community, vote that it be so; which is why you’ll notice we only ever talk about us making upgrade <em>proposals</em>.</p>
<p>Recall from <a href="https://research-development.nomadic-labs.com/meanwhile-at-nomadic-labs-12.html#the-granada-upgrade-proposal">Q2 2021</a> that:</p>
<ul>
<li><a href="https://research-development.nomadic-labs.com/florence-our-next-protocol-upgrade-proposal.html">Florence</a> was activated on <a href="https://tzstats.com/election/27">11 May 2021</a> (<a href="https://tzstats.com/1466368">block height 1,466,368</a>; <a href="https://tzstats.com/cycle/357">cycle 357</a>; <a href="https://tezos.gitlab.io/protocols/009_florence.html">changelog</a>), and</li>
<li><a href="https://forum.tezosagora.org/t/announcing-granada/3183">Granada</a> was proposed (<a href="https://tzstats.com/election/29">ongoing election</a>; <a href="https://tezos.gitlab.io/protocols/010_granada.html">changelog</a>) — and approved on 20 July.</li>
</ul>
<p>We are pleased to announce of 2021 Quarter 3 that:</p>
<ul>
<li><a href="https://research-development.nomadic-labs.com/granada-the-latest-tezos-upgrade-is-live.html">Granada went live</a> on 6 August (<a href="https://tzstats.com/election/29">vote details</a>).</li>
<li>The <a href="https://forum.tezosagora.org/t/announcing-tezos-8th-protocol-upgrade-proposal-hangzhou/3695">Hangzhou protocol upgrade proposal was released</a> on 21 September (<a href="https://tezos.gitlab.io/protocols/011_hangzhou.html">changelog</a>; <a href="https://tzstats.com/election/33">vote details</a>).</li>
<li>The Hangzhou testnet (<em>Hangzhounet</em>, of course) was released on 24 September. Joining instructions are <a href="https://www.reddit.com/r/tezos/comments/ptssy4/lets_help_testing_hangzhounet_next_protocol/">here</a>.</li>
</ul>
<p>For more information on Granada, you can see:</p>
<ul>
<li>A <a href="https://research-development.nomadic-labs.com/announcing-granada.html">detailed blogpost on the Granada upgrade</a>.</li>
<li><a href="https://research-development.nomadic-labs.com/meanwhile-at-nomadic-labs-12.html#the-granada-upgrade-proposal">The summary in Meanwhile Q2</a>.</li>
<li>Our analysis blog post on <a href="https://research-development.nomadic-labs.com/updates-granada.html">network updates from the Granada protocol amendment</a>, which is an analysis of the first complete cycle under Granada.</li>
</ul>
<p>Substantive upgrades in Hangzhou are <a href="https://tezos.gitlab.io/protocols/011_hangzhou.html">listed here</a>.
These include a new <a href="https://research-development.nomadic-labs.com/timelock-a-solution-to-minerblock-producer-extractable-value.html">Timelock primitive</a>.</p>
<h2 id="adoption-and-support"><a class="toclink" href="#adoption-and-support">Adoption and Support</a></h2>
<p>Our <a href="https://adoption-support.nomadic-labs.com/">adoption team</a> have been hard at work developing relationships, and thanks to their dilligence we are proud to report that during Q3 2021:</p>
<ul>
<li>Our adoption team supported <a href="https://en.ipocamp.io/">Ipocamp</a>, a start-up specialising in creating solutions to protect creations, in <a href="https://en.ipocamp.io/essentiel/ipocamp-choisit-tezos-pour-sa-solution-de-propriete-intellectuelle">choosing Tezos for its intellectual property solution</a> (also <a href="https://www.ipocamp.io/essentiel/ipocamp-choisit-tezos-pour-sa-solution-de-propriete-intellectuelle">in French</a>).</li>
<li>On 28 July <a href="https://www.smartlink.so/">Smartlink</a> (a decentralised escrow smart contract platform built on Tezos) <a href="https://www.smartlink.so/baker/">became a corporate Tezos baker</a>. See also <a href="https://www.nomadic-labs.com/news/smartlink-corporate-baker/">the announcement here</a>.<sup id="fnref:corporate"><a class="footnote-ref" href="#fn:corporate">2</a></sup> You can view Smartlink’s baking activity at their <a href="https://tzkt.io/tz1YiuvpfeVAHbnU8akcWErNfQGFHNQ7F45z/info">baker address</a>.</li>
<li>On 22 September <a href="https://block0.io/">Block0</a> (a blockchain consulting and development company with an emphasis on supply chain traceability and transparency) <a href="https://www.nomadic-labs.com/news/block0-corporate-baker-tezos/">became a corporate Tezos baker</a>.<sup id="fnref2:corporate"><a class="footnote-ref" href="#fn:corporate">2</a></sup> It is also the first Tezos corporate baker in Belgium.</li>
</ul>
<p>Our <a href="https://adoption-support.nomadic-labs.com">support team</a> has developed several support documents (see <a href="https://adoption-support.nomadic-labs.com/">the support homepage, ‘Useful Resources’</a> for a full list):</p>
<ul>
<li>9 July. <a href="https://adoption-support.nomadic-labs.com/wp-content/uploads/2021/07/baking-en-1.pdf">Baking: creating blocks on Tezos</a> (also <a href="https://adoption-support.nomadic-labs.com/wp-content/uploads/2021/07/baking-fr.pdf">in French</a>).</li>
<li>12 July. <a href="https://adoption-support.nomadic-labs.com/wp-content/uploads/2021/07/noeuds_en.pdf">Nodes on the Tezos blockchain</a> (also <a href="https://adoption-support.nomadic-labs.com/wp-content/uploads/2021/07/noeuds_vf.pdf">in French</a>).</li>
<li>21 July. <a href="https://adoption-support.nomadic-labs.com/wp-content/uploads/2021/07/NFT-2.pdf">NFTs on Tezos</a></li>
<li>27 July. <a href="https://adoption-support.nomadic-labs.com/wp-content/uploads/2021/07/tickets_en.pdf">Tickets on Tezos</a> (also <a href="https://adoption-support.nomadic-labs.com/wp-content/uploads/2021/07/tickets_fr.pdf">in French</a>).</li>
<li>4 August. <a href="https://adoption-support.nomadic-labs.com/wp-content/uploads/2021/07/amendement_en.pdf">Tezos Amendment Process</a> (also <a href="https://adoption-support.nomadic-labs.com/wp-content/uploads/2021/07/amendement_fr.pdf">in French</a>).</li>
<li>18 August. <a href="https://adoption-support.nomadic-labs.com/wp-content/uploads/2021/08/sapling_avec_demo.pdf">Sapling demo of confidential transactions</a> (also <a href="https://adoption-support.nomadic-labs.com/wp-content/uploads/2021/08/sapling_avec_demo-FR.pdf">in French</a>).</li>
<li>25 August. <a href="https://adoption-support.nomadic-labs.com/wp-content/uploads/2021/08/Formal_verification_ENG.pdf">Formal verification of smart contracts</a> (also <a href="https://adoption-support.nomadic-labs.com/wp-content/uploads/2021/08/Verification_Formelle.pdf">in French</a>).</li>
<li>23 September. <a href="https://adoption-support.nomadic-labs.com/wp-content/uploads/2021/09/Decentralized-Finance-on-Tezos_edited.docx.pdf">DeFi on the Tezos blockchain</a></li>
</ul>
<p>See also the list of <a href="https://research-development.nomadic-labs.com/academic.html">Tezos-related research publications</a>.</p>
<p><a name="train"></a></p>
<h2 id="training"><a class="toclink" href="#training">Training</a></h2>
<p>Training is a good thing in itself, and also a key complement to <a href="#adoption-and-support">adoption and support</a>.</p>
<p>Training in Q3 2021 proceeded apace, and we are pleased to report that we ran half-a-dozen courses training roughly a hundred people in total from countries all over the world — in European countries like Germany, France, Luxembourg, and Belgium of course; but also India, the <span class="caps">USA</span>, Canada, and Vietnam — and together with the <a href="https://tezosafrica.com/">Tezos in Africa foundation</a> we trained Tezos developers from Tunisia, Algeria, Senegal, Ghana, and Nigeria (see <a href="https://twitter.com/TezosInAfrica/status/1440993533163446272">tweet 1</a> and <a href="https://twitter.com/TezosInAfrica/status/1446595601819439106">tweet 2</a>).</p>
<p>Check out <a href="https://research-development.nomadic-labs.com/training.html">the Nomadic Labs training webpage</a>.
If you’d like us to run a training course for you, in either English or French, then you can send us an e-mail at <a href="mailto:training@nomadic-labs.com">training@nomadic-labs.com</a>.</p>
<h2 id="technical-documentation"><a class="toclink" href="#technical-documentation">Technical documentation</a></h2>
<p>Our Technical Documentation team have laboured mightily to annotate and enrich the lives of Tezos developers everywhere by expanding the <a href="https://tezos.gitlab.io/">online documentation for Tezos developers</a>.
This includes the following new content:</p>
<ul>
<li>A <a href="https://tezos.gitlab.io/developer/gadt.html">tutorial explains GADTs</a> (Generalised Algebraic Data Types) in the context of Tezos. GADTs are a hugely powerful, but complex, OCaml mechanism. The tutorial illustrates example applications of GADTs within Tezos — especially as applied to the Michelson interpreter for ensuring the absence of certain classes of smart contract runtime errors.</li>
<li>A <a href="https://tezos.gitlab.io/developer/unit_testing_legacy_code.html">tutorial</a> helps developers add unit tests to legacy code modules, by providing schemes for increasing the testability of existing code.</li>
<li>A series of pages explains <a href="https://tezos.gitlab.io/alpha/consensus.html">the consensus algorithm</a>, customised for each protocol version. As part of this revamp, existing pages on <a href="https://tezos.gitlab.io/alpha/proof_of_stake.html">Proof of Stake</a> have been restructured to describe the proof-of-stake mechanism and the associated concept of delegation, independently of the consensus algorithm.</li>
<li>We have enriched and documented <strong>event-based logging</strong>. This existing feature generalises the classic logging <span class="caps">API</span> (which will be deprecated in due course, though not just yet!). The documentation includes a <a href="https://tezos.gitlab.io/developer/event_logging_framework.html">page for developers</a> and <a href="https://tezos.gitlab.io/user/logging.html">a page for users</a>.</li>
<li>We have written structured documentation on releasing a new protocol proposal.<ul>
<li><a href="https://tezos.gitlab.io/protocols/checklist.html">The main page</a> is a checklist detailing all the steps of this process, including both technical steps and public-relation steps. Two other pages (linked to from the main page) describe specific sub-procedures:</li>
<li><a href="https://tezos.gitlab.io/developer/howto-freeze-protocols.html">Protocol freezing</a> provides a guide for removing dead code from older protocols.</li>
<li><a href="https://tezos.gitlab.io/developer/protocol_environment_upgrade.html">Adding a new environment</a> handles the case when a protocol proposal needs new features from the shell. In this case, a new protocol environment must be created. This page details all the steps involved.</li>
</ul>
</li>
<li>A new tutorial guides developers in writing and executing <a href="https://tezos.gitlab.io/developer/long-tezts.html">long-running tests in the Tezt framework</a>. This presents infrastructure for storing results in an InfluxDB database; for visualising the results with Grafana; and for sending alerts when test results differ significantly from the previous tests. Long live tests!</li>
<li>Finally, a new page in the User sections of the documentation gives an overview of the <a href="https://tezos.gitlab.io/user/versioning.html">versioning schemes</a> in use in the Tezos ecosystem: including Octez releases, protocol releases, protocol environment numbering, and <span class="caps">RPC</span> versions; and we explain which schemes are orthogonal and which are related. We hope you will find this useful.</li>
</ul>
<h2 id="phd-student-intern-and-apprentice-interviews"><a class="toclink" href="#phd-student-intern-and-apprentice-interviews">PhD student, Intern, and apprentice interviews</a></h2>
<p>We are extremely pleased at Nomadic Labs to host interns (<em>stagiaires</em>) and apprentices (<em>apprentis</em>), and to supervise some PhD students in collaboration with the local universities in Paris.
In Q2 we introduced a <a href="https://research-development.nomadic-labs.com/blog.html?category=people">‘people’ category</a> in <a href="https://research-development.nomadic-labs.com/blog.html">our blog</a>, to host interviews with our valued guests.</p>
<p>We are delighted by the variety of interesting and special people with whom we have been able to work.
Interviewees in Q3 include:</p>
<ul>
<li>Killian Delarue, an apprentice working with the Node and Tooling team on <a href="https://research-development.nomadic-labs.com/three-questions-to-nomadic-labs-apprentices-killian-delarue.html">a clean, easy and flexible way to run and monitor a Tezos node on your terminal</a>.</li>
<li>Daniel Jean, an apprentice working with the Support team to <a href="https://research-development.nomadic-labs.com/three-questions-to-nomadic-labs-apprentices-daniel-jean.html">help corporate users experiment, test, and build on the Tezos blockchain</a>.</li>
<li>Julien Coolen, an intern working with the Shell team on <a href="https://research-development.nomadic-labs.com/three-questions-to-nomadic-labs-interns-julien-coolen.html">implementing a super-scalable distributed hash table using the Octez peer-to-peer library</a>.</li>
<li>Valentin Chaboche, an intern working with the Verification and Testing teams on <a href="https://research-development.nomadic-labs.com/three-questions-to-nomadic-labs-interns-valentin-chaboche.html">lightweight property-based testing through type annotations, for the Octez codebase</a>.</li>
<li>Étienne Marais, an intern working with the Shell team on <a href="https://research-development.nomadic-labs.com/three-questions-to-nomadic-labs-interns-etienne-marais.html">energy profiling of the Tezos node</a>, with an emphasis on green computing and sustainability.</li>
<li>Mathis Gontier Delaunay, an intern working with the Umami team on <a href="https://research-development.nomadic-labs.com/three-questions-to-nomadic-labs-interns-mathis-gontier-delaunay.html">new features for the Umami wallet</a>.</li>
<li>Antonio Locascio, an intern working with the Privacy team on <a href="https://research-development.nomadic-labs.com/three-questions-to-nomadic-labs-interns-antonio-locascio.html">automatic extraction of property-based tests from F* specifications</a>.</li>
<li>Tianchi Yu, an intern working with the Michelson team on <a href="https://research-development.nomadic-labs.com/three-questions-to-nomadic-labs-interns-tianchi-yu.html">superoptimisation for the Michelson language</a>.</li>
<li>Corentin Calmels, an intern working with the Adoption team on <a href="https://research-development.nomadic-labs.com/three-questions-to-nomadic-labs-interns-corentin-calmels.html">new business solutions</a>.</li>
<li>Paul Laforgue, a PhD student working with the Verification team on <a href="https://research-development.nomadic-labs.com/five-questions-to-nomadic-labs-phds-paul-laforgue.html">specification and verification of message-passing distributed systems using choreographies</a>.</li>
</ul>
<h2 id="media-interviews-and-academic-papers"><a class="toclink" href="#media-interviews-and-academic-papers">Media Interviews and Academic Papers</a></h2>
<p>We are delighted to report that:</p>
<ul>
<li>On 7 July our adoption manager Alexia Martinel was interviewed by <a href="https://www.blockstart.eu/">BlockStart</a> about <a href="https://www.blockstart.eu/nomadic-labs-a-technical-excellence-center-to-support-the-development-of-the-tezos-ecosystem-in-france-belgium-and-luxembourg/">Tezos and what it means for France, Belgium, and Luxembourg</a>.</li>
<li>On 9 July the <a href="https://www.youtube.com/channel/UCblrftT1fOod7GsGuM8vegw">Blockchain Game Alliace</a> interviewed Alexia Martinel, our senior support engineer Florian Pautot, and others on <a href="https://www.youtube.com/watch?v=rY_6BN_UZ0g"><em>creating a gaming industry on blockchain</em></a>.</li>
<li>On 23 September Alexia Martinel spoke during the European Blockchain Week (<span class="caps">EBCW</span> 2021) in the <a href="https://www.ebcw.eu/program/">Workshop 4 on <em>Decentralized Digital Identity</em></a>. You can view <a href="https://www.ebcw.eu/video/?playlist=5336d87&video=81bfe44">the video here</a>.</li>
<li>On 13 August our <span class="caps">CEO</span> Michel Mauny was <a href="https://twitter.com/LabosNomades/status/1426566959869661185?s=20">interviewed for the 21Millions newsletter</a> (<a href="https://www.21millions.capital.fr/?utm_source=twitter-perso&utm_medium=social&utm_campaign=21Million">newsletter link</a>).</li>
<li>On 15 September our head of adoption Hadrien Zerah was interviewed in a Taleo Consulting webinar on <em>The impact of Blockchain on the Fund Industry</em>.</li>
<li>On August 11 Daniel Jean and Charles Dehlinger of Nomadic Labs gave an <a href="https://www.youtube.com/watch?v=wHW4_4VobNI">Intoduction to Tezos</a> in the <a href="https://tezasia.devfolio.co">Tezasia hackathon</a>.</li>
<li>Our support engineer Florian Pautot was a judge in the <a href="https://tezasia.devfolio.co">Tezasia hackathon</a>, which ran from 10 August to 6 September.</li>
<li>On 24 September our apprentice Daniel Jean gave <a href="https://www.youtube.com/watch?v=-A6qzTLuXrA">a talk on Day 5 of Blockchain Month Malaysia 2021</a>.</li>
</ul>
<p>We are also delighted to announce that:</p>
<ul>
<li><a href="https://arxiv.org/abs/2109.09451">Gabbay, Jakobsson, and Sojakova’s paper on the the formal <span class="caps">FA1</span>.2 ledger standard</a> was accepted to the <a href="https://fmbc.gitlab.io/2021/"><span class="caps">FMBC</span> 2021 workshop</a>. Well done team <span class="caps">GJS</span>!</li>
<li><a href="https://hal.archives-ouvertes.fr/hal-03398884">Conchon, Korneva, Bozman, Iguernlala, and Mebsout’s paper on Formally Documenting Tenderbake</a> was also accepted to the <a href="https://fmbc.gitlab.io/2021/"><span class="caps">FMBC</span> 2021 workshop</a>. Well done team <span class="caps">CKBIM</span>!</li>
</ul>
<p>We maintain a <a href="https://research-development.nomadic-labs.com/academic.html">list of Tezos-related publications</a>; if your paper should be included then please let us know at <a href="mailto:contact@nomadic-labs.com">contact@nomadic-labs.com</a>.</p>
<p>Finally, we are pleased to report on two AMAs, and on a video which we released for World Youth Skills Day on 15 July:</p>
<ul>
<li>An <a href="https://www.reddit.com/r/tezos/comments/ojni5o/tezos_community_ama_tezos_commons_nomadic_labs/"><span class="caps">AMA</span> on /r/Tezos</a> on 13 July.</li>
<li>An <a href="https://www.reddit.com/r/tezos/comments/pavhfj/tezos_community_ama_tezos_commons_nomadic_labs/"><span class="caps">AMA</span> on /r/Tezos</a> on 25 August.</li>
<li>We produced 30s of life advice for World Youth Skills Day (15 July) on <a href="https://twitter.com/i/status/1415597090890203138">tips for the next generation of software engineers</a>. The question was: <em>“What are the most important skills to build a career in the Blockchain field?”</em>. The answers included: understanding decentralisation, persistence, rigourous development practices, communication skills, abstract critical thinking, good sense of humour, rigour, strong taste for technology, humble, hard worker, and to be passionate about blockchain technology. It does seem theoretically possible for a single human being to combine all of these skills — and if you do … then please <a href="https://www.nomadic-labs.com/careers/">see our careers page</a>!</li>
</ul>
<p><em>
The following two talks by Arthur Breitman, the co-founder of Tezos, are not part of Nomadic Labs’ activities — but if you are reading this far then they well be of interest:</p>
<ul>
<li>On 5 July Arthur Breitman discussed <a href="https://www.youtube.com/watch?v=oqBSs0DSuzQ"><em>Tezos: Approaches to Scalability</em></a>.</li>
<li>On 2 August the Paul Barron network interviewed Arthur Breitman about <a href="https://www.youtube.com/watch?v=-Td0NpGcBMo"><em>upgradeable blockchain</em></a>.
</em></li>
</ul>
<h2 id="nl-research-seminars-and-blog-posts"><a class="toclink" href="#nl-research-seminars-and-blog-posts"><span class="caps">NL</span> research seminars and blog posts</a></h2>
<p>Our series of <a href="https://www.youtube.com/watch?v=2jd-1VxzmQ0&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7">Nomadic Labs research seminars</a> saw the following talks in Q3:</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=7VKeBZmEn08">Prototype of a Typical Smart Contract Agency</a> (6 July 2021).</li>
<li><a href="https://www.youtube.com/watch?v=lZfygwB743s">Specifying a Concurrent Queue in Multicore OCaml</a> (17 August 2021).</li>
<li>Then we paused for the summer. We’re French; it’s cultural. The seminars re-start in October and you can <a href="https://www.youtube.com/playlist?list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7">click here for full and up-to-date list of talks</a>.</li>
</ul>
<p>Our series of blog posts has been particularly active and you can check out <a href="https://research-development.nomadic-labs.com/blog.html?category=in-depth">our in-depth articles</a> for a growing list of detailed articles on our work.</p>
<h2 id="sponsorship"><a class="toclink" href="#sponsorship">Sponsorship</a></h2>
<p>Nomadic Labs releases all of its software and research freely and openly.
Consistent with this philosophy, we play our part in promoting an ecosystem of academic research, by contributing to the organisation of academic conferences and workshops.
In particular:</p>
<ul>
<li>We sponsored the very extensive <a href="https://qonfest2021.lacl.fr/"><span class="caps">QONFEST</span> 2021</a> federation of conferences in Paris (online) in August 2021. <span class="caps">QONFEST</span> included four well-known conferences (<a href="https://qonfest2021.lacl.fr/concur21.php"><span class="caps">CONCUR</span> 2021</a>, <a href="https://qonfest2021.lacl.fr/fmics21.php"><span class="caps">FMICS</span> 2021</a>, <a href="https://qonfest2021.lacl.fr/formats21.php"><span class="caps">FORMATS</span> 2021</a>, and <a href="https://www.qest.org/qest2021/"><span class="caps">QEST</span> 2021</a>) and also four satellite workshops (<a href="http://icetcs.ru.is/express-sos2021/">Express/<span class="caps">SOS</span> 2021</a>, <a href="https://sites.google.com/utwente.nl/pm-2021"><span class="caps">PM</span> 2021</a>, <a href="https://sites.google.com/view/snr21/"><span class="caps">SNR</span> 2021</a>, and <a href="https://concurrency-theory.org/organizations/ifip"><span class="caps">TRENDS</span> 2021</a>). Feel free to browse the <a href="https://qonfest2021.lacl.fr/">conference website</a> to see if there are any publications that might interest you.</li>
<li>We sponsored <a href="https://fmbc.gitlab.io/2021/"><span class="caps">FMBC</span> 2021</a>, the 3rd International Workshop on Formal Methods for Blockchains in July 2021. You can download the <a href="https://fmbc.gitlab.io/2021/files/FMBC2021.pdf"><span class="caps">FMBC</span> 2021 preproceedings here</a> (<a href="https://web.archive.org/web/20211025084911/https://fmbc.gitlab.io/2021/files/FMBC2021.pdf">permalink</a>).</li>
</ul>
<h2 id="contract-calls"><a class="toclink" href="#contract-calls">Contract calls</a></h2>
<p>Nomadic Labs is part of the Tezos ecosystem, and on this topic it may be worth noting that the number of smart contract calls on Tezos has displayed a striking exponential increase in Q3 2021.</p>
<p>Here are the figures in brief:
Tezos turned three years old on 30 June 2021 (see the <a href="https://tzstats.com/0">genesis block, baked on 30 June 2018</a>).
In those first three years we had 5 million contract calls.
Then we saw <a href="https://twitter.com/madfishofficial/status/1417715364465713156?s=20">another 5 million contract calls in July and August</a> — and then <em>another</em> 5 million contract calls <em>just in September</em>.
This means that Tezos saw twice as much smart contract activity in Q3 2021, as it saw in the entire first three years of its existence, so this quarter does appear to have witnessed a step change in activity.</p>
<p>You can view <a href="https://better-call.dev/stats/mainnet/general">summary bar charts here</a>.
Nomadic Labs is pleased to have played its part in these encouraging and excellent developments.</p>
<p><a name="prochaine"></a></p>
<h2 id="a-la-prochaine"><a class="toclink" href="#a-la-prochaine">À la prochaine</a></h2>
<p>Before you go, we’d like to advertise:</p>
<ul>
<li>The <a href="https://www.infrachainsummit.com/">Infrachain summit</a> on 18 November in Luxembourg, in Italy, and Online. Infrachain aims to support and foster the European blockchain industry and the Summit is about supporting real business with a gathering dedicated to blockchain topics.</li>
<li>Nomadic Labs will have a booth at the <a href="https://www.digitalfinancesummit.com/">Digital Finance Summit 2021</a> in Brussels, with an overarching theme of Sustaining the Economic Recovery.</li>
</ul>
<p>We hope to see you there.</p>
<p>Thanks for reading what we’ve been up to in Quarter 3 of 2021: three months of Nomadic Labs building and testing software and extending public understanding and adoption of blockchain technology.
Do check in again for the next Meanwhile for Quarter 4 of 2021.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:deep">
<p>Computer scientists would call this is a <em>deep embedding</em> of Michelson in Coq. <a class="footnote-backref" href="#fnref:deep" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:corporate">
<p>A <strong>corporate baker</strong> is a corporate institution that sets up one or more block validating nodes on the Tezos blockchain. <a href="https://tezos.gitlab.io/011/glossary.html#baker">More information is on bakers is here</a>. <a class="footnote-backref" href="#fnref:corporate" title="Jump back to footnote 2 in the text">↩</a><a class="footnote-backref" href="#fnref2:corporate" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>A Deep Dive into the Octez Prevalidator2021-10-25T14:00:00+02:002021-10-25T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-10-25:/a-deep-dive-into-the-octez-prevalidator.html<p>In this deep dive into the waters of the Octez shell, we will describe recent work towards improving the prevalidator. As we plunge in, we will visit how operation classification works, why the propagation of consensus of operations is critical — and the effort it took to get this right.</p><!-- This article has been crafted using, and spell-checked against,
an `enUS` dictionary -->
<p>In this blog post we’ll describe recent work on improving the <a href="https://gitlab.com/tezos/tezos">Tezos Octez</a> <a href="http://tezos.gitlab.io/shell/validation.html#prevalidator"><em>prevalidator</em></a> by making it faster and more resilient, and outline our plans for the future.</p>
<ul>
<li><a href="#a-brief-map-of-tezos-situating-the-prevalidator">A brief map of Tezos, situating the prevalidator</a></li>
<li><a href="#updating-the-prevalidator">Updating the prevalidator</a><ul>
<li><a href="#how-we-had-to-">How we had to …</a></li>
<li><a href="#-because-of-consensus-operations">… because of consensus operations</a></li>
</ul>
</li>
<li><a href="#a-primer-on-tezos-prevalidation">A primer on Tezos Prevalidation</a><ul>
<li><a href="#the-mempool-data-structure">The mempool data structure</a></li>
<li><a href="#initial-processing-to-pending-or-not-to-pending">Initial processing: to Pending or not to Pending</a></li>
<li><a href="#further-classification-of-pending-operations-into-reservoirs">Further classification of Pending operations into reservoirs</a></li>
<li><a href="#flushing-the-reservoirs">Flushing the reservoirs</a></li>
</ul>
</li>
<li><a href="#why-the-propagation-of-consensus-operations-should-be-as-fast-as-possible">Why the propagation of consensus operations should be as fast as possible</a></li>
<li><a href="#an-operation-on-tezos-from-the-shells-point-of-view">An operation on Tezos (from the shell’s point of view)</a></li>
<li><a href="#propagation-of-endorsements-arriving-too-early">Propagation of endorsements arriving too early</a></li>
<li><a href="#making-the-prevalidator-harder">Making the prevalidator <em>harder</em></a></li>
<li><a href="#testing-the-prevalidator">Testing the prevalidator</a></li>
<li><a href="#the-future-of-the-octez-prevalidator">The future of the Octez prevalidator</a></li>
</ul>
<h2 id="a-brief-map-of-tezos-situating-the-prevalidator"><a class="toclink" href="#a-brief-map-of-tezos-situating-the-prevalidator">A brief map of Tezos, situating the prevalidator</a></h2>
<p>Let’s outline how the prevalidator fits into the Tezos blockchain workflow:</p>
<ol>
<li>The Tezos blockchain is a chain of blocks — going right back to the <a href="https://tzstats.com/0"><em>genesis block</em></a> on 30 June 2018 at just after 5pm.</li>
<li>Each <strong>block</strong> is a sequence of <strong>operations</strong> (e.g. “Credit 1 tez from my account to yours”; “credit 5 tez from my account to the supermarket”; “grab this nifty <span class="caps">NFT</span> from my favorite marketplace”; …)</li>
<li>Users propose operations, which are hashed, and the hashes are bundled into small packages called <strong>mempools</strong>, which are gossiped<sup id="fnref:gossip"><a class="footnote-ref" href="#fn:gossip">1</a></sup> across the Tezos peer-to-peer (p2p) network.</li>
<li>The Tezos <strong>prevalidator</strong> is a software component that mediates between the peer-to-peer (p2p) network communications layer (which gossips information across the network) and the Tezos <strong>economic protocol</strong> (which decides which operations are valid and thus potential candidates for inclusion in blocks on the blockchain).</li>
<li>A Tezos <strong>baker</strong> packages operations up into blocks, and then adds the block to the chain.</li>
<li>The Tezos <strong>consensus algorithm</strong> does the work of organizing consensus across the network on which blocks get included in the final, official blockchain history.</li>
</ol>
<p>In a little more detail, the prevalidator does several things:</p>
<ul>
<li>It scans incoming mempool data for unknown operation hashes, and if it sees one such hash then it may request the full contents of that operation.</li>
<li>It may invoke the economic protocol to assess the <em>prevalidity</em> of operations, and accordingly <a href="#classify">store them in appropriate reservoirs</a>.</li>
<li>It stores known (but not-yet-included-in-blocks) operations as <a href="#a-primer-on-tezos-prevalidation">described below</a>.</li>
<li>It may pass operations back to the p2p layer for distribution, with or possibly without a prevalidity check.</li>
<li>A Tezos baker may dip into the prevalidator’s reservoir of prevalidated operations, looking for operations to bake into new blocks.</li>
</ul>
<p>To paraphrase <a href="https://en.wikipedia.org/wiki/Animal_Farm">Animal Farm</a>: every part of the Tezos blockchain is equally critical, but the prevalidator is more critical than most.
So it’s important that the Tezos Octez prevalidator should be correct, efficient, and resilient.</p>
<h2 id="updating-the-prevalidator"><a class="toclink" href="#updating-the-prevalidator">Updating the prevalidator</a></h2>
<h3 id="how-we-had-to"><a class="toclink" href="#how-we-had-to">How we had to …</a></h3>
<p>Since the Octez code went live with <a href="https://research-development.nomadic-labs.com/meanwhile-at-nomadic-labs-11.html#announcing-1000-days-of-tezos-mainnet">the launch of the Tezos <em>mainnet</em> over 3 years ago</a>, we have tended to avoid changing the prevalidator’s code, except for bug-fixes and some code linting.</p>
<p>The prevalidator is a critical component and an error in it could crash the network.
It ain’t broke, so we didn’t try to fix it.
Nevertheless, while the prevalidator has served us well, it does need to be improved.</p>
<p>A <a href="https://gitlab.com/tezos/tezos/-/merge_requests/968">prototype re-implementation of the prevalidator</a> was considered, but we decided it would be safer to make careful, incremental improvements to the existing code base, and since April 2020 we have undertaken significant efforts to
modernize its implementation and (just as importantly) to increase the prevalidator’s test coverage.</p>
<p>One of the key functions of the Octez prevalidator is to make sure that consensus operations — <strong>endorsements</strong><sup id="fnref:consensus-ops"><a class="footnote-ref" href="#fn:consensus-ops">2</a></sup> — are propagated sufficiently quickly and efficiently, even in adverse conditions.
This can be subtle because it has to do with the behavior of complex interacting systems, and this turns out to be a rich source of failure modes of the Tezos blockchain overall.
Regrettably, the propagation of endorsements has been affected by a number of bugs recently, which became evident “in the wild” after <a href="https://research-development.nomadic-labs.com/granada-the-latest-tezos-upgrade-is-live.html">activation of the Granada protocol</a> which reduced the minimal block time from 60s to 30s.</p>
<p>It may seem counter-intuitive that making the time between blocks <em>shorter</em> could make the network overall <em>slower</em> — but this is the joy of designing blockchain systems.
We will explain how this works below (a key point is <a href="#increase">here</a>).</p>
<p>Fixing these bugs has motivated several recent releases of Octez: notably <a href="https://tezos.gitlab.io/releases/version-9.html">v9.2, v9.4 and v9.7</a>, and to a lesser degree, v9.6.</p>
<h3 id="because-of-consensus-operations"><a class="toclink" href="#because-of-consensus-operations">… because of consensus operations</a></h3>
<p><em>Propagation of consensus operations</em> will be the particular focus of this article, because it is these operations’ particular interaction with the prevalidator that caused us issues over the summer.</p>
<p>We hope this article will explain what we are doing to get this right, and will shed some technical light on what happened with the Tezos network last summer, specifically during the
<a href="https://research-development.nomadic-labs.com/updates-granada.html#block-times">activation of the Granada protocol</a>.
We will also describe current and future endeavors to make the prevalidator <a href="https://www.youtube.com/watch?v=yydNF8tuVmU&ab_channel=DaftPunk"><em>Harder, Better, Faster, Stronger!</em></a></p>
<p>Or to put it another way: we’ll explain what went wrong, what role the prevalidator played in this, and what we’re doing to update the prevalidator to ensure that this particular failure mode will not be repeated.</p>
<h2 id="a-primer-on-tezos-prevalidation"><a class="toclink" href="#a-primer-on-tezos-prevalidation">A primer on Tezos Prevalidation</a></h2>
<h3 id="the-mempool-data-structure"><a class="toclink" href="#the-mempool-data-structure">The mempool data structure</a></h3>
<p>On the Tezos blockchain, operations are gossiped using a container
data structure called <strong>mempool</strong>, which can roughly<sup id="fnref:mempool"><a class="footnote-ref" href="#fn:mempool">3</a></sup> be
described as a collection of operation hashes. These operation hashes
are advertised in a single <code>CurrentHead</code> network message which contains</p>
<p><a id="single"></a></p>
<ul>
<li>a block header — which is usually the one from the head of the Tezos chain as seen by the sender node when it broadcasts its current head — and</li>
<li>a mempool data structure containing the hashes of some operations that the sender node wants to broadcast.</li>
</ul>
<h3 id="initial-processing-to-pending-or-not-to-pending"><a class="toclink" href="#initial-processing-to-pending-or-not-to-pending">Initial processing: to <code>Pending</code> or not to <code>Pending</code></a></h3>
<p>On reception of this message, a Tezos node running the Octez Tezos implementation does two things:</p>
<ul>
<li>it passes the block header to its <a href="http://tezos.gitlab.io/shell/validation.html#validator">block validator</a> — the shell component in charge of block validation — and</li>
<li>it passes the mempool to its prevalidator.</li>
</ul>
<p>The prevalidator processes the incoming mempool, finds any hashes which are unknown to the receiving node, and sends a request to the sender node for the full content of the corresponding operations.
This saves on bandwidth, since network hashes are far more compact than full operations.</p>
<p>The prevalidator performs some basic checks on a new operation that it is well-formed according to the currently-active economic protocol: for example it parses the operation and checks that the data is not gibberish and that the operation would pay enough fees.
(What it does not do at this stage is apply the larger and more expensive <code>apply_operation</code> method from the economic protocol; see next paragraph.)
If the incoming operation passes these basic checks, then it is marked as <code>Pending</code> by the prevalidator.</p>
<h3 id="further-classification-of-pending-operations-into-reservoirs"><a class="toclink" href="#further-classification-of-pending-operations-into-reservoirs">Further classification of <code>Pending</code> operations into reservoirs</a></h3>
<p>Operations which have passed basic checks to become <code>Pending</code> operations, are collected in a set of operations<sup id="fnref:set"><a class="footnote-ref" href="#fn:set">4</a></sup> called <code>pending</code> — so <code>pending</code> is the set of operations with <code>Pending</code> status — that are next in line to be <em>prevalidated</em> using the <code>apply_operation</code> method of the economic protocol.<sup id="fnref:pre"><a class="footnote-ref" href="#fn:pre">5</a></sup>
To ensure fairness with respect to the other components of the node, pending operations are prevalidated in batches of fixed maximal size.</p>
<p><a id="classify"></a></p>
<p>The result returned by <code>apply_operation</code> enables the prevalidator to classify each operation in a batch as follows:<sup id="fnref:apply"><a class="footnote-ref" href="#fn:apply">6</a></sup></p>
<ol>
<li>
<p><code>Applied</code>: the operation can be applied in the current <strong>context</strong> (meaning: the ledger state, as
seen by the node in question).
However, the application of the operation might still fail
later, in the event a baker decides to include the operation in a block.</p>
</li>
<li>
<p><code>Branch delayed</code>: the operation cannot be included in the next
head of the chain, but it could be included in a descendant.
Example: a <em>manager operation</em><sup id="fnref2:manager"><a class="footnote-ref" href="#fn:manager">7</a></sup> with a counter value in
the future of the expected one.</p>
</li>
<li>
<p><code>Branch refused</code>: the operation cannot be included in the next
head of the chain, nor in a successor, but it might be applied on
a different branch if a reorganization happens. Example: a
manager operation whose counter value is in the past of the
current counter.</p>
</li>
<li>
<p><code>Refused</code>: the operation is impossible and should be rejected. It
is not valid, and there is no alternative chain in which it might
be. Example: an operation with an invalid signature.</p>
</li>
</ol>
<p>The prevalidator stores the operations in different reservoirs<sup id="fnref:pun"><a class="footnote-ref" href="#fn:pun">8</a></sup> according to this classification.</p>
<h3 id="flushing-the-reservoirs"><a class="toclink" href="#flushing-the-reservoirs">Flushing the reservoirs</a></h3>
<p>Regularly — which in normal operation of the <a href="https://forum.tezosagora.org/t/announcing-granada/3183">current economic protocol</a> means every 30 seconds — the node’s chain will change (e.g. by adding another block), and the block validator will update its head block.
This means that the current block state has changed so that prevalidity of operations — and the
ensuing classification — in the current state of the prevalidator
might be outdated and need to be revised.
The node triggers the prevalidator to perform a <strong>flush</strong> event (also called a <strong>recycling</strong> event) in which:</p>
<ul>
<li>each operation classified as <code>Applied</code> or <code>Branch delayed</code> and not yet included in a block is reverted to a <code>Pending</code> state, and</li>
<li>each operation classified as <code>Branch refused</code> is reverted to a
<code>Pending</code> state, provided that the new head is <strong>not</strong> a direct
successor of the previous one; that is, if the node switched
branches — otherwise they remain classified as <code>Branch refused</code>.</li>
</ul>
<p>These newly pending operations should then be re-prevalidated.<sup id="fnref:coined"><a class="footnote-ref" href="#fn:coined">9</a></sup></p>
<p>And the cycle repeats …</p>
<p>Our description so far has been a high-level, somewhat idealized account of the prevalidator’s execution logic.
It omits some details, including protections and counter-measures against DDoS attacks from malicious peers.
One of these protections will feature in what follows, because it impacts the propagation of consensus operations and will turn out to be fundamental to our story.</p>
<h2 id="why-the-propagation-of-consensus-operations-should-be-as-fast-as-possible"><a class="toclink" href="#why-the-propagation-of-consensus-operations-should-be-as-fast-as-possible">Why the propagation of consensus operations should be as fast as possible</a></h2>
<p>The Tezos protocol has relied to date on Proof of Stake consensus algorithms from the <a href="https://research-development.nomadic-labs.com/emmy-seven-years-of-updatable-consensus.html">Emmy family</a>.
The currently-active consensus protocol is Emmy*, which has been running on-chain since the <a href="https://research-development.nomadic-labs.com/granada-the-latest-tezos-upgrade-is-live.html">activation of the Granada protocol</a> (at <a href="https://tzstats.com/protocols/PtGRANADsDU8R9daYKAgWnQYAJ64omN1o3KMGVCykShA97vQbvV">block 1,589,248 on 6 August 2021</a>).</p>
<p><a name="well-branched"></a></p>
<p>Consensus algorithms <em>à la Emmy</em> come with a special kind of operation
called an <strong>endorsement</strong>, whose purpose is to facilitate reaching
consensus faster. <a href="https://research-development.nomadic-labs.com/emmy-seven-years-of-updatable-consensus.html#endorsements">This blog
post</a>
digs deeper into how consensus algorithms work and the role
endorsements play in effectively securing the Tezos blockchain. The
details relevant to us here are:</p>
<ul>
<li>
<p>An endorsement operation included in a block at level <span class="math">\(n+1\)</span>,<sup id="fnref:levels"><a class="footnote-ref" href="#fn:levels">10</a></sup>
endorses the predecessor block at level <span class="math">\(n\)</span>. Hence, a baker cannot
include an endorsement in its block without having seen and
validated its predecessor block at level <span class="math">\(n\)</span>.</p>
</li>
<li>
<p>The <strong>minimal block delay</strong> (the protocol-mandated time gap between two consecutive blocks at
levels <span class="math">\(n\)</span> and <span class="math">\(n + 1\)</span>)<sup id="fnref:bd"><a class="footnote-ref" href="#fn:bd">11</a></sup> depends on the number of endorsements
included in the block at level <span class="math">\(n + 1\)</span>.
With Emmy*, the time between two blocks is <span class="math">\(30\)</span> seconds for a
priority <span class="math">\(0\)</span> block that contains at least <span class="math">\(60\%\)</span> of all possible
<span class="math">\(256\)</span> endorsements — that is <span class="math">\(154\)</span> endorsements. With (say) <span class="math">\(150\)</span>
endorsements, the delay escalates to <span class="math">\(60 + 4\cdot (192-150) = 228\)</span> seconds — nearly <span class="math">\(4\)</span> minutes!</p>
</li>
</ul>
<p>Thus including more endorsements in a block means</p>
<ul>
<li>a shorter minimal block delay (i.e. the block can be baked earlier), and so</li>
<li>a greater chance that this block (rather than any competing blocks from other block producers) will get included in the final blockchain history, and also</li>
<li>higher <a href="http://tezos.gitlab.io/active/consensus.html#rewards">block rewards payed to the baker</a> if the block gets included in the final blockchain history.</li>
</ul>
<p><a id="increase"></a></p>
<p>If endorsements are validated and propagated quickly across the
network, from endorsers to bakers, then blocks can be produced with optimal speed.
And as a corollary: it is <em>very important that endorsements arrive promptly relative to the minimal block delay</em>, because if endorsements consistently fail to arrive in time for the <em>minimal</em> block delay then — <em>by design</em> — the block delays as imposed by the consensus algorithm may substantially increase.</p>
<p>And this was how <em>reducing</em> the minimal block delay could potentially make the network <em>slower</em>, if the endorsements start arriving “late” relative to this new minimal block delay.
Thus, as we reduce the minimal block delay in search of greater speed and efficiency, it becomes increasingly important that we expedite the passage of endorsements across the network.
This in turn requires us to tweak the prevalidator to recognize and prioritize endorsement operations, and — such is the reality of programming complex systems — this requires much careful and precise work which we will now discuss.</p>
<h2 id="an-operation-on-tezos-from-the-shells-point-of-view"><a class="toclink" href="#an-operation-on-tezos-from-the-shells-point-of-view">An operation on Tezos (from the shell’s point of view)</a></h2>
<p>The Tezos protocol distinguishes between</p>
<ul>
<li>the <em>(economic) protocol</em>, and</li>
<li>the <em>shell</em>.</li>
</ul>
<p>This is summed up in <a href="http://tezos.gitlab.io/shell/the_big_picture.html">a well-known octopus diagram</a> in the Tezos architecture description.<sup id="fnref:octopus"><a class="footnote-ref" href="#fn:octopus">12</a></sup></p>
<ul>
<li>The economic protocol defines the blockchain protocol’s execution logic (consensus algorithms, what kinds of operations are available, smart contracts, …). This can be updated and is subject to <em>self-amendment</em>.</li>
<li>The shell handles the lower level, more <em>mundane</em> tasks: p2p gossiping, context storage, maintaining a distributed database of known blocks and operations.
And — key to our story today — the shell triggers the validation of blocks and the prevalidation of operations, by choosing blocks and operations and applying appropriate validation functions from the economic protocol.</li>
</ul>
<p>The prevalidator is a component of the shell.
It makes calls to functions from the economic protocol to help it decide whether operations are valid — but the prevalidator itself is agnostic about precisely how the economic protocol decides.</p>
<p>This agnosticism implies that the Octez suite must implement the shell parametrically over the economic protocol and in such a way that it can deal seamlessly with updates to the protocol.
One particular design consequence of this is that the shell must take a rather high-level and generic view of the internal content of certain data structures like operations and blocks, so that there will be an abstract and generic view for the shell, and then a specific view for each specific economic protocol.<sup id="fnref:eop"><a class="footnote-ref" href="#fn:eop">13</a></sup> Thus, from the shell’s perspective, we have <a href="https://gitlab.com/tezos/tezos/-/blob/master/src/lib_base/operation.mli">roughly
the following
</a> (<a href="https://archive.softwareheritage.org/swh:1:cnt:e805835834386cf320d619fc5d987fde14814a9a;origin=https://gitlab.com/tezos/tezos/;visit=swh:1:snp:87ee09fa0e69f6f51488e931f0635f7f890a755c;anchor=swh:1:rev:f7a88f3a80a222e17d6b191f67bfc8993e7d6390;path=/src/lib_base/operation.mli">permalink</a>):</p>
<div class="highlight"><pre><span></span><code><span class="c">(* The branch of an operation is the block hash of the block upon which</span>
<span class="c"> the operation was forged. *)</span>
<span class="k">type</span> <span class="n">shell_header</span> <span class="o">=</span> <span class="o">{</span><span class="n">branch</span> <span class="o">:</span> <span class="nn">Block_hash</span><span class="p">.</span><span class="n">t</span><span class="o">}</span>
<span class="c">(* An operation from the shell point of view is a branch and a list of bytes.</span>
<span class="c"> Only the economic protocol knows how to interpret those bytes. *)</span>
<span class="k">type</span> <span class="n">t</span> <span class="o">=</span> <span class="o">{</span>
<span class="n">shell</span> <span class="o">:</span> <span class="n">shell_header</span><span class="o">;</span> <span class="n">proto</span> <span class="o">:</span> <span class="nn">Bytes</span><span class="p">.</span><span class="n">t</span><span class="o">}</span>
</code></pre></div>
<p>The <code>branch</code> of an operation is then a block hash.
It determines the lifespan of an operation: if the block hash is <em>too old</em>, the operation can be discarded.
The current lifespan on Emmy* is <span class="math">\(120\)</span> blocks,<sup id="fnref:max_op_ttl"><a class="footnote-ref" href="#fn:max_op_ttl">14</a></sup> which (assuming a normal 30s per block) translates to about one human hour.</p>
<p>For the prevalidator, the branch field is also used as an anti-spam filter: if the block hash is <em>unknown</em>, then the operation can be safely discarded.
In this context, unknown means “a block that has not been validated yet”.</p>
<p>The <code>proto</code> field is then — in the prevalidator’s eyes — a mere list
of <code>Bytes</code>. The economic protocol has the key to unlock the secret of
whether that operation is an endorsement or something else. And this
detail is central to the plot of the next section.</p>
<h2 id="propagation-of-endorsements-arriving-too-early"><a class="toclink" href="#propagation-of-endorsements-arriving-too-early">Propagation of endorsements arriving too early</a></h2>
<p>Granada (the currently active economic protocol) and also the <a href="https://marigold.dev/blog/announcing-hangzhou">Hangzhou
protocol proposal</a>,
require that the branch field of an endorsement operation point to
the block being endorsed.
This requirement is due to a pre-existing legacy format
for endorsements,<sup id="fnref:branch"><a class="footnote-ref" href="#fn:branch">15</a></sup> but the requirement can hinder the propagation of endorsements by the prevalidator, in the following corner case:</p>
<p>Suppose that the node receives a new <code>CurrentHead</code> message from the network: this <a href="#single">consists of a block header and a mempool</a>.
Suppose that this block header is not known to the receiving node, and suppose that the mempool includes operations which are endorsements of that very block.
In this corner case, it may be that the prevalidation of the received mempool completes earlier than the validation of the received block (even if only by a second or two).<sup id="fnref:scratch"><a class="footnote-ref" href="#fn:scratch">16</a></sup>
The block header only needs to be validated once, when it is first seen — but this is that first time.
In this case, any endorsements for that block that may be contained in the mempool message, will be discarded and not propagated, since the branch of the endorsement is <em>precisely</em> the not-quite-yet-validated block.</p>
<p>Other kinds<sup id="fnref:kinds"><a class="footnote-ref" href="#fn:kinds">17</a></sup> of operations, like manager operations, avoid this
problem by not branching on the immediate predecessor, but rather on
an ancestor block three or four levels behind.</p>
<p><a name="v9.2-patch"></a></p>
<p>Addressing the race between the validator and the prevalidator in a
scenario of simultaneously receiving a new block header and an
endorsement operation for that same block, required small patches to the shell and to the economic protocol:</p>
<ul>
<li>
<p><em>On the shell side,</em> the prevalidator now asks the economic protocol whether the freshly-received
operation is a valid endorsement, before checking if the operation was
<em>branched</em> on a known block — that is, before checking if the block
header targeted by the <code>branch</code> field of the endorsement operation is
indeed a member of the set of recent <code>live_blocks</code> known to the
node. In this case the prevalidator classifies the endorsement as
<code>Branch refused</code>, and triggers the advertisement of its current head
and of a mempool including the not-yet-validated newly-found endorsement.</p>
</li>
<li>
<p><em>On the protocol side,</em> the <code>apply_operation</code> function now checks whether the signature of an endorsement has been verified, <em>before</em> checking whether its branch is known. This avoids propagating endorsements with bad signatures (i.e. it avoids sending spam).</p>
</li>
</ul>
<p>These changes were both included in <a href="http://tezos.gitlab.io/CHANGES.html#version-9-2">Octez v9.2</a>, which in
addition to the new version of the node also included the “protocol
snapshot” for Granada, effectively advertising to the network (and to
bakers in particular) that there was a protocol candidate which could
be injected.</p>
<p>After the release of Octez v9.2, fewer endorsements were missed. However, <a href="#v9.2-patch">the small patches mentioned above</a> created two unforeseen issues:<sup id="fnref:chronology"><a class="footnote-ref" href="#fn:chronology">18</a></sup></p>
<ol>
<li>
<p>The network could propagate endorsements that were <em>too old</em> and were already included in previous blocks (more than <span class="math">\(60\)</span> blocks ago by then, <span class="math">\(120\)</span> nowadays<sup id="fnref2:max_op_ttl"><a class="footnote-ref" href="#fn:max_op_ttl">14</a></sup>).
This is wasteful, as these operations would be <code>Refused</code> by the receiving peer’s prevalidator.</p>
</li>
<li>
<p>The execution cost of checking the signature of an endorsement, in
the case that the endorsement’s level <em>is not the one expected</em>, is
significantly higher than the execution cost of checking the signature
of a <em>well-branched</em> endorsement (meaning one included <a href="#well-branched">in the
successor block to its target</a>). This is because only
the public keys of endorsers for the current level are cached by the
economic protocol. For the rest, checking their signatures requires a
hard-drive read, which may be computationally expensive.</p>
</li>
</ol>
<p>The first issue was fixed in v9.4, and the second issue — which is
what caused nodes to slow down when Granada was <a href="https://tzstats.com/BLzEHPYwpqWSDfu28saTkAUxZ7WnPkNxe8Epg5NfixBfQv7KRUN">activated on August
6th 2021 at 09:36
<span class="caps">UTC</span></a>
— was fixed with v9.7.<sup id="fnref2:chronology"><a class="footnote-ref" href="#fn:chronology">18</a></sup></p>
<p>The idea behind the fixes is this: in Octez, there are filters specific to an economic
protocol which can classify an operation based on its content. We
tweaked those filters to only consider endorsements which are at the
appropriate level.</p>
<p>In order to prevent such issues from recurring, we have structured our work on the prevalidator around answering three questions:</p>
<ol>
<li>How can we test the prevalidator beyond integration tests?</li>
<li>How can we reliably benchmark the prevalidator?</li>
<li>How can we make the prevalidator faster?</li>
</ol>
<p>We tackle test coverage first, then benchmarking, and then speed, because: test coverage catches functional regressions, benchmarking permits us to profile and measure speed, and then we are in a position to both measure performance and more safely change code to improve it.</p>
<p><a id="harder"></a></p>
<p>Thus during the summer of 2021 (and learning from the experience with the activation of Granada) we focused on the first task above, which we call <strong>hardening</strong> the prevalidator — or to fit the story, <a href="https://gitlab.com/tezos/tezos/-/milestones/16"><em>making it harder</em></a>.
We aim to tackle the remaining two tasks during autumn 2021.</p>
<h2 id="making-the-prevalidator-harder"><a class="toclink" href="#making-the-prevalidator-harder">Making the prevalidator <em>harder</em></a></h2>
<p><a id="first"></a></p>
<p>The prevalidator is part of the shell as we mentioned above — but it calls functions from the economic protocol to parse operations and validate them, so that it can decide whether to propagate them.
This interaction between the shell and the economic protocol, and how this separation is implemented in the codebase, is fruitful but it <em>does</em> make the prevalidator tricky to test — especially with unit tests.</p>
<p>Hence, the first stage of our work was to isolate the interface of the prevalidator with the economic protocol, by refactoring the prevalidator to separate</p>
<ul>
<li>those components and those parts of the execution logic that just have to do with the shell, from</li>
<li>those components that are in the shell but interact directly with the economic protocol.</li>
</ul>
<p>This refactoring enabled us to identify and fix a number of minor bugs, including: several memory leaks, and some minor corner cases where valid operations were not being propagated.</p>
<p>This was nice, but the main achievement of this work was to isolate
the <em>core</em> of the prevalidator: a software component, reified as an
<a href="https://gitlab.com/tezos/tezos/-/blob/master/src/lib_shell/prevalidator_classification.mli">OCaml module</a>,
which classifies operations correctly.<sup id="fnref:modules"><a class="footnote-ref" href="#fn:modules">19</a></sup></p>
<h2 id="testing-the-prevalidator"><a class="toclink" href="#testing-the-prevalidator">Testing the prevalidator</a></h2>
<p>Any sufficiently large software project should employ multiple testing frameworks to provide increased confidence in the implementation.
This is especially true for a complex project like the Tezos Octez implementation, each of whose different components — from the lower-level grit of the distributed network to the higher-level Michelson interpreter — is an unruly beast of software in its own right, just waiting to spring bugs and regressions on the unsuspecting software engineer.</p>
<p>Of the many testing frameworks available to <em>the working Tezos dev</em>,<sup id="fnref:testing"><a class="footnote-ref" href="#fn:testing">20</a></sup>
two are particularly relevant to our story here:</p>
<ul>
<li>
<p><strong>Unit</strong> and <strong>Property Based Tests</strong> for the Prevalidator using,
respectively, the <a href="https://github.com/mirage/alcotest">Alcotest</a>
and <a href="https://github.com/c-cube/qcheck">QCheck</a> OCaml testing libraries.</p>
</li>
<li>
<p><strong>Integration tests</strong> using
<a href="https://tezos.gitlab.io/developer/tezt.html">Tezt</a>, a custom-made
testing framework for system and integration tests for Tezos,
focusing on the interaction between Octez nodes and clients.</p>
</li>
</ul>
<p>Prior to April 2021 there were few unit tests for the prevalidator because of the technical issues described in the <a href="#first">first paragraph of this section</a>.
Our refactoring allowed us to be more rigorous and to add many unit tests for the prevalidator, enabling us to detect and fix legacy bugs.
Most of the new unit tests are moreover <em>property-based tests</em>: they not only assert the prevalidator invariant, but also test that the root cause of certain bugs is no longer there.</p>
<p>An example of the latter property is the following:</p>
<blockquote>
<p><em>Given any operation, at any time, an operation cannot be classified twice with two different classifications</em>.</p>
</blockquote>
<p>That is, an operation cannot be at the same time <code>Applied</code> and <code>Branch refused</code>.</p>
<p>In summary we have added:</p>
<ul>
<li>18 property-based tests asserting correctness properties of the classification of operations, and</li>
<li>12 integration tests covering different scenarios in the prevalidator’s life-cycle.</li>
</ul>
<p>Of the latter, an important integration test we have included asserts
that consensus operations which arrive too early are still propagated
by the node.</p>
<h2 id="the-future-of-the-octez-prevalidator"><a class="toclink" href="#the-future-of-the-octez-prevalidator">The future of the Octez prevalidator</a></h2>
<p>After <a href="#harder">making the prevalidator <em>harder</em></a>, the next step is to make it
<a href="https://gitlab.com/tezos/tezos/-/milestones/39"><em>better</em></a>.
To this end we aim to enhance the prevalidator, working towards two main objectives:</p>
<ol>
<li>
<p>We aim to minimize the elapsed time between when an operation is
received (either injected via <span class="caps">RPC</span> from a client, or gossiped from
the network), and when we propagate the operation to our peers.</p>
</li>
<li>
<p>We aim to prioritize the handling of pending operations to give more priority to pending <em>consensus</em> operations, and thus prevalidate them as quickly as possible (the current prevalidator treats all pending operations equally).</p>
</li>
</ol>
<p>The first objective will help improve operation throughput — increasing the <span class="caps">TPS</span> metric (transactions per second).
This will require non-trivial changes to the economic protocol, which we hope to include in the forthcoming protocol proposal (for <a href="https://gitlab.com/tezos/tezos/-/milestones/43">protocol I</a>).</p>
<p>The second objective will empower the node to discard operations sooner, if their priority is too low.
This will help ensure that critical operations (e.g. endorsements, and consensus operations in general, and also high-fee operations) will be retained by the prevalidator and propagated more effectively.</p>
<p>Looking further into the future, making Octez scale to cope with the
<a href="https://twitter.com/tezos/status/1432417900653162502?s=20">ongoing</a>
<a href="https://xtz.news/latest-tezos-news/tezos-passes-650000-transactions-in-24-hours-and-1-7-million-contract-calls-in-10-days">increase</a>
in <a href="https://better-call.dev/stats/mainnet/general">network traffic</a> will require ever more efficient validation and propagation time for
operations and blocks, and this will require a comprehensive effort
across the Octez architecture.</p>
<p>Our work so far has created a <em>harder</em> prevalidator, as a foundation to build a <em>better one</em>.
<em>Faster</em> and <em>stronger</em> will follow.<sup id="fnref:hfs"><a class="footnote-ref" href="#fn:hfs">21</a></sup>
Stay tuned!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:gossip">
<p><em>Gossip</em> here is a technical p2p term for
”(unsolicited) broadcast to a subset of known peers”. Gossiping
contributes to global network coverage of knowledge — like an
office rumor (or an epidemic). <a class="footnote-backref" href="#fnref:gossip" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:consensus-ops">
<p>Tezos is a <em>Proof of Stake</em> blockchain which
requires participants to vote for block candidates. These votes,
called <em>endorsements</em>, are included in blocks and recorded on-chain, just as “regular” <em>transactions</em> are. We <a href="#why-the-propagation-of-consensus-operations-should-be-as-fast-as-possible">expand on this below</a>. <a class="footnote-backref" href="#fnref:consensus-ops" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:mempool">
<p>The <a href="https://gitlab.com/tezos/tezos/-/blob/master/src/lib_base/mempool.mli#L26">current implementation</a> (<a href="https://archive.softwareheritage.org/swh:1:cnt:9550fcb2c9a050214632b0eeac550d935ed9bc74;origin=https://gitlab.com/tezos/tezos/;visit=swh:1:snp:4e9bc213eb79ae105dc144dffa8aa92d627b041c;anchor=swh:1:rev:990a58a126fa4f2f5bfc7dc42c36e8277dc3112a;path=/src/lib_base/mempool.mli;lines=26">permalink</a>)
is a record, with two fields: a <code>known_valid</code> list of operation
hashes, presumably prevalidated by the node; and a <code>pending</code> set
consisting of operations of <em>unknown</em> prevalidity, or
invalid-yet-safe-to-broadcast operations. <a class="footnote-backref" href="#fnref:mempool" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:set">
<p>We mean <em>set</em> here in the mathematical sense: an
<em>unsorted</em>, collection with <em>unique</em> members. The
current specification does not impose an order on the set of pending
operations. However, given the nature of the implementation with
<code>pending : Operation.t Operation_hash.Map.t</code>, in practice iteration
over <code>pending</code> follows the lexicographical order on operation hashes. <a class="footnote-backref" href="#fnref:set" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:pre">
<p>In Tezos we refer to this phase as <strong>operation prevalidation</strong> to distinguish it from the validation of operations done by the baker when producing a new block, and by a node’s (block) validator when receiving a new block.
The entry point to the economic protocol is the same in both cases: the <code>apply_operation</code> method from the protocol <span class="caps">API</span>. <a class="footnote-backref" href="#fnref:pre" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:apply">
<p>The <code>apply_operation</code> function is so called because it checks
whether it can apply the operation to the current ledger state (as
seen by the node on which this is running) — in Tezos jargon,
this ledger state is called the <strong>(blockchain) context</strong>. If
<code>apply_operation</code> succeeds, it returns an <code>Ok</code> value; if it cannot
do this, then it returns an <code>Error</code> value <a href="https://tezos.gitlab.io/developer/error_monad.html#the-actual-tezos-error-monad">of a certain
kind</a>.
This returned value enables the
prevalidator to further classify each operation depending on
whether it could be included in the next head of the chain, on the
same branch later on, or possibly on some other branch, or never. <a class="footnote-backref" href="#fnref:apply" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:manager">
<p><em>Manager operations</em> include transfers, smart contract
originations, calls to smart contracts, etc. In short, any
<em>fee-paying</em> operation in competition for block space. Manager
operations have a <em>counter</em> associated (unsurprisingly) to their
manager, which is the Tezos account which signs the operation and
which pays the fees. <a class="footnote-backref" href="#fnref:manager" title="Jump back to footnote 7 in the text">↩</a><a class="footnote-backref" href="#fnref2:manager" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn:pun">
<p>The pun with “mempool” is deliberate. Readers diving into this text with a low tolerance for wordplay can fish out the word “reservoir”, and splash in the word “container” instead, and they will get on swimmingly. <a class="footnote-backref" href="#fnref:pun" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
<li id="fn:coined">
<p><span class="dquo">“</span>Reprevalidated”: this blog post’s contribution to the English language. <a class="footnote-backref" href="#fnref:coined" title="Jump back to footnote 9 in the text">↩</a></p>
</li>
<li id="fn:levels">
<p>The <em>level</em> of a block is simply how many blocks are between it and the genesis block (which is at level 0). <a class="footnote-backref" href="#fnref:levels" title="Jump back to footnote 10 in the text">↩</a></p>
</li>
<li id="fn:bd">
<p>The minimal block delay is defined as a function of a block’s priority and the number of
included endorsements. See the <a href="http://tezos.gitlab.io/active/consensus.html#minimal-block-delay-function">full definition in the documentation</a>. <a class="footnote-backref" href="#fnref:bd" title="Jump back to footnote 11 in the text">↩</a></p>
</li>
<li id="fn:octopus">
<p>This is also the <em>motif</em> in a limited series of coveted laptop stickers. <a class="footnote-backref" href="#fnref:octopus" title="Jump back to footnote 12 in the text">↩</a></p>
</li>
<li id="fn:eop">
<p>You can read <a href="https://research-development.nomadic-labs.com/the-protocol-from-high-level-command-line-to-low-level-operations.html">this article</a> for a holistic account of the life cycle of operations, including the perspective from inside the economic protocol. <a class="footnote-backref" href="#fnref:eop" title="Jump back to footnote 13 in the text">↩</a></p>
</li>
<li id="fn:max_op_ttl">
<p>The constant regulating this behavior is called
<code>max_op_ttl</code> (<em>“maximal operation time-to-live”</em>). It was
bumped from <span class="math">\(60\)</span> to <span class="math">\(120\)</span> when Granada was activated, to
compensate for the halving in minimal block time and thus keeping
operation lifespan (approximately) constant. <a class="footnote-backref" href="#fnref:max_op_ttl" title="Jump back to footnote 14 in the text">↩</a><a class="footnote-backref" href="#fnref2:max_op_ttl" title="Jump back to footnote 14 in the text">↩</a></p>
</li>
<li id="fn:branch">
<p>In it, the <code>proto</code> field contained only the level
endorsed. The rationale behind this design was to save space, as
endorsements must be branched in the block immediately following the
target. More compact endorsements means more operations inside
blocks. And that seemed a clever win at the time. <a class="footnote-backref" href="#fnref:branch" title="Jump back to footnote 15 in the text">↩</a></p>
</li>
<li id="fn:scratch">
<p>You may be wondering how the block validator can validate a block
if it just has its header. The trick is that the
<code>block_header</code> carries a lot of information (unlike the <code>branch</code>
of operations), including the hash of the resulting context and
the hash of the Merkle tree of operations. By design, a block
header and the full contents of its operations, is all the information that the block validator
actually needs. More on this in <a href="https://tezos.gitlab.io/shell/validation.html">the documentation for the validation subsystem</a>. <a class="footnote-backref" href="#fnref:scratch" title="Jump back to footnote 16 in the text">↩</a></p>
</li>
<li id="fn:kinds">
<p>From genesis to the latest Tezos protocol there have been four
<em>kinds</em> (or <em>classes</em>) of operations: <em>consensus</em> (e.g.,
endorsements), <em>voting</em> (e.g., protocol injection and ballots),
<em>anonymous</em> (e.g., double-baking or double-endorsement
accusations), and the <em>manager</em> operations described above.<sup id="fnref:manager"><a class="footnote-ref" href="#fn:manager">7</a></sup> <a class="footnote-backref" href="#fnref:kinds" title="Jump back to footnote 17 in the text">↩</a></p>
</li>
<li id="fn:chronology">
<p>A brief chronology is as follows: Octez v9.2 with the <a href="#v9.2-patch">aforementioned patches</a> and advertisment of Granada; then 9.3 and 9.4; then Granada activation on August 6th; then Octez v9.5, 9.6, and 9.7. <a class="footnote-backref" href="#fnref:chronology" title="Jump back to footnote 18 in the text">↩</a><a class="footnote-backref" href="#fnref2:chronology" title="Jump back to footnote 18 in the text">↩</a></p>
</li>
<li id="fn:modules">
<p>That is, we isolated the core of the execution logic of the prevalidator, and reified this as an OCaml module called <a href="https://gitlab.com/tezos/tezos/-/blob/master/src/lib_shell/prevalidator_classification.mli"><code>prevalidator_classification</code></a> — in fact, this is technically just a data structure, albeit one containing an <span class="caps">API</span>, functions, and internal data structures. OCaml is an object-oriented functional programming language in which modules are first-class, so this kind of encapsulation is practical and all in a good day’s work. If you’re reading this and it blows your mind, but in a good way, then feel free to be in touch to discuss employment opportunities. <a class="footnote-backref" href="#fnref:modules" title="Jump back to footnote 19 in the text">↩</a></p>
</li>
<li id="fn:testing">
<p>More details can be found on <a href="https://tezos.gitlab.io/developer/testing_index.html">Testing in
Tezos</a> — the
entry point in the Tezos Developer Documentation for all things tests. <a class="footnote-backref" href="#fnref:testing" title="Jump back to footnote 20 in the text">↩</a></p>
</li>
<li id="fn:hfs">
<p>If you liked this blog post you may also like <a href="https://research-development.nomadic-labs.com/a-tale-of-two-reductions-in-gas-consumption-in-tezos.html">a previous blog post on optimizing gas consumption</a>. It describes a distinct set of optimizations in a distinct part of the Tezos universe, but it is interesting to note a resemblance in the overall workflow, described there as <a href="https://research-development.nomadic-labs.com/a-tale-of-two-reductions-in-gas-consumption-in-tezos.html#conclusion">“make it work, make it right, make it fast”</a>. Thank you for reading. <a class="footnote-backref" href="#fnref:hfs" title="Jump back to footnote 21 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Introducing a new Storage Backend for Octez2021-10-18T16:00:00+02:002021-10-18T16:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-10-18:/introducing-a-new-storage-backend-for-octez.html<p>A ligher, faster, better storage system for the Tezos Octez blockchain implementation</p><p>A <em>blockchain</em> is a chain of blocks, where a block is a blob of data holding relevant transactions and other data.
This means that <a href="https://gitlab.com/tezos/tezos/">Octez</a> — the Tezos blockchain implementation to which Nomadic Labs <a href="https://research-development.nomadic-labs.com/announcing-octez.html">contributes</a> — should optimize how it handles this particular data structure.</p>
<p>The recently-released <a href="http://tezos.gitlab.io/releases/version-10.html">Version 10 of Octez</a> introduces a new storage backend for Octez, with new features and performance improvements (referenced as <code>v0.0.6</code> in <a href="http://tezos.gitlab.io/releases/version-10.html#changelog">the Changelog</a>).</p>
<p>In this blog post we will outline what these optimizations are, and how they work.</p>
<h2 id="octez-nodes-store-in-a-nutshell"><a class="toclink" href="#octez-nodes-store-in-a-nutshell">Octez node’s store in a nutshell</a></h2>
<p>The storage layer of an Octez node has two parts:</p>
<ul>
<li><strong>the context</strong>, containing the ledger state as a git-like versioned repository, and</li>
<li><strong>the store</strong>, containing the blocks, operations and protocols of the chain.</li>
</ul>
<p>Our Version 10 update is to the store part; context storage is unchanged.</p>
<p>The previous implementation relied on the generic key-value database <a href="http://www.lmdb.tech/doc/"><span class="caps">LMDB</span></a>.
This works, but because it is generic it cannot accomodate optimizations that (in the Tezos use-case) would increase performance.</p>
<p>Therefore, we implemented a bespoke storage layer to optimize read/write speed, memory usage, storage size, and concurrent access.</p>
<h3 id="the-key-insight"><a class="toclink" href="#the-key-insight">The key insight</a></h3>
<p><span class="caps">LMDB</span> introduces a global lock when writing new data to the store.
It has to because (in the general case) it cannot know how a write might affect the database.</p>
<p>Global locks are expensive, and in the Tezos use-case they are mostly unnecessary: intuitively, blockchains consist mostly of <em>immutable data</em> — except for forks on recent blocks as discussed below — so we know that fresh writes will <em>not</em> affect data if it is old enough, and in fact only a small lock on some concurrent read accesses is required.</p>
<p>We need a little background before we can flesh out the particulars of how to turn this insight into an optimized implementation:</p>
<h2 id="some-background-on-time-and-consensus"><a class="toclink" href="#some-background-on-time-and-consensus">Some background on time and consensus</a></h2>
<ul>
<li>In Tezos, time is split into <em>cycles</em>, where currently 1 cycle = 8192 blocks ≈ 68 hours.<sup id="fnref:time"><a class="footnote-ref" href="#fn:time">1</a></sup></li>
<li>Tezos reaches global <em>consensus</em> about the state of the chain in a few levels, under normal network conditions.<sup id="fnref:consensus"><a class="footnote-ref" href="#fn:consensus">2</a></sup> To have a very generous safety margin, Tezos considers that there cannot exist a fork that is longer than 5 cycles ≈ 14 days.</li>
</ul>
<p>Thus the blockchain can be treated conceptually as consisting of</p>
<ul>
<li>a long chain of immutable (<em>cemented</em>) blocks which come before the <strong>no fork point</strong>, and for which consensus is certain, and</li>
<li>a recent and shorter chain of possibly mutable blocks which come strictly after the <strong>no fork point</strong>, for which consensus (<em>finality</em>) is uncertain.</li>
</ul>
<p>We illustrate this with a graphic:</p>
<p align="center" style="width=100%">
<img width="100%" src="images/new_store_backend/architecture.png" alt="The new store architecture"/>
</p>
<p>This new structure allows us to optimize the store, as we discuss next.</p>
<p><a id="new-store"></a></p>
<h2 id="the-new-store-structure"><a class="toclink" href="#the-new-store-structure">The new store structure</a></h2>
<h3 id="cemented-store"><a class="toclink" href="#cemented-store">Cemented store</a></h3>
<p><strong>Cemented store</strong> manages cemented cycles (left of the <em>no fork point</em> illustrated above), which contain cemented blocks. A cemented cycle is a contiguous list of blocks which reflects the same notion of cycles from the economic protocol.
These are immutable so they can have a fast ‘cemented look-up’:</p>
<p><a id="canonical"></a></p>
<ul>
<li>We assume finality for cemented blocks, so history is <em>canonical</em> in the sense that every node has the same cemented storage representation.</li>
<li>The block metadata of each cemented block can be compressed, to reduce storage size.<sup id="fnref:compression"><a class="footnote-ref" href="#fn:compression">3</a></sup></li>
<li>We maintain two indexes that allow for rapid block accesses, increasing the overall process performance: a <code>hash -> level</code> index, and its converse <code>level -> hash</code>. These are cheap to implement and maintain, because we know they need not be updated once they have been populated.</li>
</ul>
<h3 id="floating-store"><a class="toclink" href="#floating-store">Floating store</a></h3>
<p>The floating store splits blocks further into two sub-stores:</p>
<ul>
<li>a <strong>Read Only</strong> (<span class="caps">RO</span>) floating store of blocks, and</li>
<li>a <strong>Read Write</strong> (<span class="caps">RW</span>) floating store.</li>
</ul>
<p>The <span class="caps">RO</span> store is <em>read-only</em>; no blocks are added, removed, or updated.
The <span class="caps">RW</span> store is <em>append-only</em>; no blocks are removed or updated, but blocks may be added.</p>
<p>A quick summary of why we have this distinction:</p>
<ul>
<li>
<p>The <span class="caps">RO</span> store contains any blocks left over from the floating store of the previous <a href="#merge">merge procedure</a> that were neither <a href="#cement">cemented</a> nor <a href="#trim">trimmed</a>. Details are <a href="#merge">below</a>.</p>
</li>
<li>
<p>The <span class="caps">RW</span> store contains any blocks that have arrived since the previous merge procedure was <a href="#initiate">initiated</a>.</p>
</li>
<li>
<p>The floating store (<span class="caps">RO</span>+<span class="caps">RW</span>) contains blocks which are candidates for reorganization or trimming. It may contain forks and consensus is not final.</p>
</li>
</ul>
<p>We now come to the merge procedure, whereby blocks in <span class="caps">RO</span>+<span class="caps">RW</span> are examined, organized, trimmed, and (where appropriate) shifted from the floating to the cemented store:</p>
<p><a id="merge"></a></p>
<h3 id="the-merge-procedure"><a class="toclink" href="#the-merge-procedure">The merge procedure</a></h3>
<p>When shifting the <strong>no fork point</strong> forward,<sup id="fnref:nfp"><a class="footnote-ref" href="#fn:nfp">4</a></sup> we move some blocks from the floating store to the cemented store using a <strong>merge procedure</strong>, as follows:</p>
<ol>
<li><strong>Initiate</strong> the merge procedure by locking <span class="caps">RW</span> so no further appends are accepted. <a id="initiate"></a> To ensure liveness during and after the the merge procedure, create an empty <span class="caps">RW</span>’ store to store blocks as they continue to arrive.</li>
<li><strong>Retrieve</strong> the blocks between the <strong>previous no fork point</strong> (Previous <span class="caps">NFP</span>) and the <strong>new no fork point</strong> (New <span class="caps">NFP</span>), starting from the head. This corresponds to the cycle(s) that we might cement.</li>
<li><a id="trim"></a> <strong>Trim</strong> branches in the selected cycle(s) to those that remain valid according to <a href="https://research-development.nomadic-labs.com/emmy-seven-years-of-updatable-consensus.html">the consensus algorithm</a>.</li>
<li><a id="cement"></a> <strong>Cement</strong> the blocks for which we have consensus and place them in a new cemented cycle.</li>
<li><strong>Combine</strong> any leftover blocks — meaning blocks in <span class="caps">RO</span>+<span class="caps">RW</span> that were not deleted by the trimming procedure, but for which consensus is uncertain so that they have not made it into the cemented store — into a new <span class="caps">RO</span>’ store.</li>
<li><strong>Promote</strong> the <span class="caps">RO</span>’ and <span class="caps">RW</span>’ as the new <span class="caps">RO</span> and <span class="caps">RW</span> stores.</li>
</ol>
<p align="center" style="width=100%">
<img width="100%" src="images/new_store_backend/merge.png" alt="The merge procedure"/>
</p>
<h2 id="performance-benchmarks"><a class="toclink" href="#performance-benchmarks">Performance benchmarks</a></h2>
<p><em>These benchmarks were computed on a mid-range laptop with <span class="caps">SSD</span> storage running a Tezos node with Mainnet data (which at time of benchmarking had approximately <code>1.65</code> million blocks).</em></p>
<h3 id="store-overall-performance"><a class="toclink" href="#store-overall-performance">Store overall performance</a></h3>
<p>With the bespoke store format <a href="#new-store">discussed above</a>, our new storage backend significantly reduces the node’s disk I/O requirements, memory consumption, and storage footprint.</p>
<p>We can benchmark the storage footprint below as follows:</p>
<ul>
<li>The block history maintained by a Mainnet node in full mode now requires around <code>8GB</code> of disk space, compared to more than <code>26GB</code> before.</li>
<li>The block history required in archive mode now requires <code>20GB</code> of disk space, compared to between <code>40GB</code> and <code>60GB</code>.</li>
</ul>
<p align="center" width=85%>
<img width=55% src="images/new_store_backend/store_size.png" alt="Storage footprint size"/>
</p>
<h3 id="snapshot-overall-performance"><a class="toclink" href="#snapshot-overall-performance">Snapshot overall performance</a></h3>
<p>Thanks to the new snapshot version and new store format — moving from the <code>v.0.0.1</code> (in Octez <code><= 9.x</code>) to <code>v.0.0.2</code> (in Octez <code>>=v10.x</code>) — snapshots are lighter and faster to import and export.
Indeed, snapshots are now based on the <em>canonical</em> layer of the store, which allows more optimizations:</p>
<ul>
<li>Snapshot <strong>export time</strong> is improved by 1.8x, and snapshot <strong>import time</strong> is improved by 3.7x.</li>
<li>Snapshot <strong>size</strong>, for full mode, is reduced by around 20%.</li>
</ul>
<p align="center" width=100%>
<img width=45% src="images/new_store_backend/export.png" alt="Snapshot export time"/>
<img width=45% src="images/new_store_backend/import.png" alt="Snapshot import time"/>
<img width=45% src="images/new_store_backend/snapshot_size.png" alt="Snapshot size"/>
</p>
<h2 id="new-features-of-the-storage-backend"><a class="toclink" href="#new-features-of-the-storage-backend">New features of the storage backend</a></h2>
<p>The new storage backend of <a href="http://tezos.gitlab.io/releases/version-10.html">Octez version 10</a> is lighter and faster as discussed above.
It is also more resilient and contains numerous other improvements and tweaks.
We conclude with a brief changelog:</p>
<ol>
<li>New snapshot format (<code>v2</code>) is lighter and faster to export and import and can be exported in two formats:<ul>
<li>(default) as a <code>tar</code> archive: easy to exchange,</li>
<li>as a <code>raw</code> directory: suitable for <a href="https://ipfs.io"><span class="caps">IPFS</span> exchange</a> thanks to the <a href="#canonical">canonical representation of the block history</a>.</li>
</ul>
</li>
<li>A new command <code>tezos-node snapshot info</code> allows to inspect the information of a given snapshot.</li>
<li>Rolling mode <a href="http://tezos.gitlab.io/user/history_modes.html#history-modes">is now labelled as <code>rolling mode</code></a>, rather than <code>experimental-rolling</code>.</li>
<li>Reworked history modes to be able to configure a cycle offset to preserve more cycles, if needed. By default, the node now retains 5 cycles with metadata below the <em>no fork point</em> (i.e. <code>full+5</code>), instead of 3 cycles (i.e. <code>full+3</code>).</li>
<li>The store is more resilient thanks to a consistency check and an automatic restore procedure to recover from a corrupted state, meaning that if (for example) power to your node cuts, you are more likely to be able to just reboot and carry on.</li>
</ol>
<p>Further details of new features, and information on how to upgrade, are in the <a href="http://tezos.gitlab.io/releases/version-10.html#changelog">full Octez <code>v10</code> changelog</a>.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:time">
<p>At time of writing Tezos is running the <a href="http://tezos.gitlab.io/protocols/010_granada.html">Granada protocol</a>, in which a block lasts 30s during typical, normal operation. Thus, 1 block = 30s and 1 cycle = just over 68 hours. Since this blog post is written from the point of view of Tezos, we will take blocks and cycles as our primitive notion of time henceforth (rather than seconds and hours). You know you think about blockchain too much, when you tell your child “hurry up to bed, it’s just 20 blocks till bedtime”. <a class="footnote-backref" href="#fnref:time" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:consensus">
<p>A discussion of <a href="https://research-development.nomadic-labs.com/emmy-seven-years-of-updatable-consensus.html">Nakamoto-style consensus is here (based on Emmy*)</a> and one of <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">classical <span class="caps">BFT</span>-style consensus is here (based on Tenderbake)</a>. For the purposes of this blog post, all that matters is that there are blocks, and consensus is reached. <a class="footnote-backref" href="#fnref:consensus" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:compression">
<p>Compressing data reduces storage size but introduces overhead since accessing the data requires some computation to decompress it. However, the cemented blocks are infrequently accessed so for cemented store, this tradeoff is worthwhile. <a class="footnote-backref" href="#fnref:compression" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:nfp">
<p><em>When</em> the no fork point moves, and how far, is governed by constants and algorithms in <a href="http://tezos.gitlab.io/protocols/010_granada.html">the current economic protocol</a>. In practice the economic protocols so far just choose constants such that the fork point gets updated to remain relatively recent, yet far enough into the past for consensus to be certain. <a class="footnote-backref" href="#fnref:nfp" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
</ol>
</div>Improving the implementation of cryptography in Tezos Octez2021-10-14T13:00:00+02:002021-10-14T13:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-10-14:/improving-the-implementation-of-cryptography-in-tezos-octez.html<p>Recent developments in the <span class="caps">HACL</span>* / EverCrypt crypto toolstack in the Tezos Octez implementation</p><hr>
<ul>
<li><a href="#1-our-cryptography-toolchain-and-why-it-matters">1. Our cryptography toolchain, and why it matters</a></li>
<li><a href="#2-the-hacl-library">2. The <span class="caps">HACL</span>* library</a></li>
<li><a href="#21-quick-correct-and-compatible-pick-three">2.1 Quick, correct, and compatible: pick three</a></li>
<li><a href="#22-three-cryptographic-primitives">2.2 Three cryptographic primitives</a><ul>
<li><a href="#221-p-256">2.2.1 P-256</a></li>
<li><a href="#222-sha-3">2.2.2 <span class="caps">SHA</span>-3</a></li>
<li><a href="#223-blake2">2.2.3 <span class="caps">BLAKE2</span></a></li>
</ul>
</li>
<li><a href="#3-evercrypt-an-api-for-hacl-and-more-">3. EverCrypt: an <span class="caps">API</span> for <span class="caps">HACL</span>* (and more)</a></li>
<li><a href="#31-how-evercrypt-enhances-hacl-">3.1 How EverCrypt enhances <span class="caps">HACL</span>*</a></li>
<li><a href="#32-advantages-of-having-a-cryptographic-provider">3.2 Advantages of having a cryptographic provider</a></li>
<li><a href="#33-the-ocaml-api">3.3 The OCaml <span class="caps">API</span></a></li>
<li><a href="#4-a-deep-dive-into-the-ocaml-api">4. A deep dive into the OCaml <span class="caps">API</span></a></li>
<li><a href="#41-the-low-level-api-hacl-star-raw-">4.1 The low-level <span class="caps">API</span> <code>hacl-star-raw</code></a></li>
<li><a href="#42-the-high-level-api-hacl-star-">4.2 The high-level <span class="caps">API</span> <code>hacl-star</code></a></li>
<li><a href="#43-using-the-high-level-api">4.3 Using the high-level <span class="caps">API</span></a></li>
<li><a href="#5-further-reading">5. Further reading</a></li>
<li><a href="#acknowledgements">Acknowledgements</a></li>
</ul>
<h2 id="1-our-cryptography-toolchain-and-why-it-matters"><a class="toclink" href="#1-our-cryptography-toolchain-and-why-it-matters">1. Our cryptography toolchain, and why it matters</a></h2>
<p>A safety-critical program is only as trustworthy as the libraries it relies on, so we at Nomadic Labs pay close attention to our tools and dependencies — i.e. to <strong>our toolchain</strong>.</p>
<p>Our toolchain is based on <a href="https://hacl-star.github.io/"><span class="caps">HACL</span>*</a>; a verified library of cryptographic primitives, which include
the hash functions which are the backbone of blockchain technology (principally: <a href="https://www.blake2.net/">Blake2</a>), and
the digital signatures which we use to assure transaction authenticity (<a href="https://en.wikipedia.org/wiki/EdDSA#Ed25519">Ed25519</a>, <a href="https://web.archive.org/web/20210403195235/https://asecuritysite.com/encryption/ecdsa_p256">P-256</a>, and <a href="https://en.bitcoin.it/wiki/Secp256k1">secp256k1</a> <sup id="fnref:3"><a class="footnote-ref" href="#fn:3">2</a></sup>).</p>
<p>In this blog post, we’ll discuss recent improvements to our cryptography toolchain, and how we integrated them into practical OCaml programming of the <a href="https://gitlab.com/tezos/tezos/">Tezos Octez implementation</a>. Namely:</p>
<ul>
<li><span class="caps">HACL</span>* has been enriched and improved with new crypto primitives, and</li>
<li>access to those primitives has been improved by introducing a sophisticated new <a href="#cryptoprovider">cryptographic provider</a> called <a href="https://hacl-star.github.io/HaclValeEverCrypt.html">EverCrypt</a>.</li>
</ul>
<p>In this blog post, we’ll survey</p>
<ul>
<li>what the <span class="caps">HACL</span>* crypographic library offers,</li>
<li>how the cryptographic provider EverCrypt can enhance it, and</li>
<li>the scaffolding we use to efficiently and reliably invoke these powerful tools from within the Tezos Octez implementation.</li>
</ul>
<p><a name="hacl"></a></p>
<h2 id="2-the-hacl-library"><a class="toclink" href="#2-the-hacl-library">2. The <span class="caps">HACL</span>* library</a></h2>
<h3 id="21-quick-correct-and-compatible-pick-three"><a class="toclink" href="#21-quick-correct-and-compatible-pick-three">2.1 Quick, correct, and compatible: pick three</a></h3>
<p><a href="https://hacl-star.github.io/"><span class="caps">HACL</span>*</a> is a cryptographic library<sup id="fnref:0"><a class="footnote-ref" href="#fn:0">1</a></sup> offering a comprehensive collection of cryptographic primitives (we give <a href="#three">three examples below</a>).</p>
<p><span class="caps">HACL</span>* is complex, safety-critical code.
Accordingly, it is written in and formally verified using the <a href="https://www.fstar-lang.org/">F* language</a>, and then extracted (compiled) to correct and efficient C using <a href="https://github.com/FStarLang/kremlin">KreMLin</a>, a tool which translates a subset of F* to C.
KreMLin also facilitates using the generated C code in OCaml projects — such as the Tezos Octez implementation — by automatically building <a href="#OCaml-API">the scaffolding that developers would otherwise write themselves</a>.</p>
<p>The resulting code offers three <em>formal safety guarantees</em>:
<a id="formal-safety-guarantees"></a></p>
<ol>
<li><strong>Functional correctness:</strong> the code’s behaviour complies with its specification.</li>
<li><strong>Memory safety:</strong> memory is managed correctly — so no buffer overflows, dereferencing null pointers, accessing memory after it has been freed, etc.</li>
<li><strong>Secret independence:</strong> the C instructions executed, in what order, and any memory accesses, do not depend on any secret values. This protects against <a href="https://en.wikipedia.org/wiki/Timing_attack">timing attacks</a> that might slurp unintended information.</li>
</ol>
<p>For more details see the <a href="https://eprint.iacr.org/2017/536">original <span class="caps">HACL</span>* paper</a>.</p>
<p>The point of this toolstack is that it gives us three important properties which it is far from trivial to reconcile:</p>
<ol>
<li>Safety guarantees as described above, while</li>
<li>retaining the state-of-the-art performance required of a cryptographic library in a real-world system, and</li>
<li>safely interfacing with the code from within OCaml.</li>
</ol>
<p><a id="three"></a></p>
<h3 id="22-three-cryptographic-primitives"><a class="toclink" href="#22-three-cryptographic-primitives">2.2 Three cryptographic primitives</a></h3>
<p>The Tezos Octez implementation has been using <span class="caps">HACL</span>* since before its launch,
and Nomadic Labs has been actively supporting the continued development of <span class="caps">HACL</span>*: via grants to the <a href="prosecco.gforge.inria.fr/">Prosecco team</a> at Inria Paris; and through the work of engineers at Nomadic.</p>
<p><span class="caps">HACL</span>* offers implementations for all but one<sup id="fnref2:3"><a class="footnote-ref" href="#fn:3">2</a></sup> of the core crytographic primitives used by the Nomadic Labs <a href="https://gitlab.com/tezos/tezos/">Octez</a> implementation of Tezos.</p>
<p>Let’s survey three examples, which are cryptographic primitives recently introduced into <span class="caps">HACL</span>* and relevant to the Tezos implementation:</p>
<h4 id="221-p-256"><a class="toclink" href="#221-p-256">2.2.1 P-256</a></h4>
<p>P-256 (also called secp256r1) is an <a href="https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm">elliptic curve</a> signature algorithm and one of the three signature schemes supported in Tezos: P-256; Ed25519; and secp256k1.
P-256 is a <a href="http://www.secg.org/sec2-v2.pdf"><span class="caps">NIST</span> standard</a> with wide industry support.
It allows interoperability with HSMs (hardware security modules), including hardware wallets and Apple’s Secure Enclave.</p>
<!-- https://web.archive.org/web/20210228075416/https://www.johndcook.com/blog/2018/08/21/a-tale-of-two-elliptic-curves/ -->
<p>Any Tezos address generated with P-256 starts with <code>tz3</code>.</p>
<p>A <a href="https://github.com/project-everest/hacl-star/tree/master/code/ecdsap256">verified implementation of P-256</a> by the Prosecco team is now in <span class="caps">HACL</span>* and replaces our previous library.</p>
<h4 id="222-sha-3"><a class="toclink" href="#222-sha-3">2.2.2 <span class="caps">SHA</span>-3</a></h4>
<p>Version 1 of the <a href="https://tezos.gitlab.io/developer/protocol_environment.html">Tezos protocol environment</a> <a href="https://gitlab.com/tezos/tezos/-/commit/49eff245f081539a4b9ec10e5827eb29d7379965">introduced</a> three hash functions based on the <a href="https://en.wikipedia.org/wiki/SHA-3">Keccak algorithm</a>:</p>
<ul>
<li><span class="caps">SHA3</span>-256 and <span class="caps">SHA3</span>-512 from the official <a href="https://csrc.nist.gov/publications/detail/fips/202/final"><span class="caps">NIST</span> <span class="caps">SHA</span>-3 standard</a>, and</li>
<li>Keccak256, another variation of the Keccak algorithm, which is <a href="https://ethereum.org/en/glossary/#keccak-256">the hash function used in Ethereum</a>.</li>
</ul>
<p>Starting with the <a href="https://research-development.nomadic-labs.com/edo-the-latest-tezos-upgrade-is-live.html">Edo protocol upgrade</a>, these three hash functions are available as Michelson opcodes,
alongside the previously-present hash functions <span class="caps">BLAKE2</span> and <span class="caps">SHA</span>-256 and <span class="caps">SHA</span>-512 (which are two versions of <span class="caps">SHA</span>-2).</p>
<p><a name="blake-2"></a></p>
<h4 id="223-blake2"><a class="toclink" href="#223-blake2">2.2.3 <span class="caps">BLAKE2</span></a></h4>
<p><a href="https://www.blake2.net/"><span class="caps">BLAKE2</span></a> is the main hash function of the Tezos protocol.
<span class="caps">BLAKE2</span> hashes everything from individual keys and messages, to whole blocks.<sup id="fnref:2bornot2b"><a class="footnote-ref" href="#fn:2bornot2b">3</a></sup></p>
<p>We now use <span class="caps">BLAKE2</span> via its new <span class="caps">HACL</span>* implementation.
This gives us the three <a href="#formal-safety-guarantees">formal safety guarantees above</a>, and furthermore <span class="caps">HACL</span>* offers</p>
<ul>
<li>a portable C implementation of <span class="caps">BLAKE2</span> which runs on any platform, and</li>
<li>a faster <em>vectorized</em> implementation which assumes Intel’s <a href="https://en.wikipedia.org/wiki/Advanced_Vector_Extensions">Advanced Vector Extensions 2 (<span class="caps">AVX2</span>)</a>, which offer <span class="caps">CPU</span> instructions for <a href="https://en.wikipedia.org/wiki/SIMD"><span class="caps">SIMD</span></a> (single instruction, multiple data) parallelism.</li>
</ul>
<p>Modern cryptographic algorithms, including <span class="caps">BLAKE2</span>, are often designed from the ground up to allow implementors to make certain optimizations where possible, such as using hardware features like <span class="caps">AVX2</span> or <span class="caps">SIMD</span> mentioned above where the hardware supports this.
See a <a href="https://eprint.iacr.org/2020/572">paper on how this fits into <span class="caps">HACL</span>*’s verification pipeline</a>: for example, the <span class="caps">SIMD</span> implementation is around 30% faster than the non-<span class="caps">SIMD</span> one.<sup id="fnref:4"><a class="footnote-ref" href="#fn:4">4</a></sup></p>
<h2 id="3-evercrypt-an-api-for-hacl-and-more"><a class="toclink" href="#3-evercrypt-an-api-for-hacl-and-more">3. EverCrypt: an <span class="caps">API</span> for <span class="caps">HACL</span>* (and more)</a></h2>
<h3 id="31-how-evercrypt-enhances-hacl"><a class="toclink" href="#31-how-evercrypt-enhances-hacl">3.1 How EverCrypt enhances <span class="caps">HACL</span>*</a></h3>
<p>Having a great library like <span class="caps">HACL</span>* is one thing.
But how to package its features for developers to efficiently deploy in working code?
This is where EverCrypt can help.</p>
<p>EverCrypt is a cryptographic provider that bundles the cryptographic primitives present in <span class="caps">HACL</span>* into a unified package.
EverCrypt — which like <span class="caps">HACL</span>* is written in F* and extracted to correct and efficient C using <a href="https://github.com/FStarLang/kremlin">KreMLin</a> — does not replace <span class="caps">HACL</span>* so much as provide an additional interface to access the suite of cryptographic primitives offered by <span class="caps">HACL</span>*. EverCrypt also bundles routines written in highly-optimized assembly verified with a tool called <a href="https://github.com/project-everest/vale">Vale</a>.<sup id="fnref:caveat"><a class="footnote-ref" href="#fn:caveat">5</a></sup></p>
<p><a id="cryptoprovider"></a></p>
<p>We can sum this up as follows:</p>
<ul>
<li><span class="caps">HACL</span>* is a <em>cryptographic library</em>.</li>
<li>EverCrypt is a <em>cryptographic provider</em>.</li>
</ul>
<h3 id="32-advantages-of-having-a-cryptographic-provider"><a class="toclink" href="#32-advantages-of-having-a-cryptographic-provider">3.2 Advantages of having a cryptographic provider</a></h3>
<p>In practice, accessing cryptographic primitives through EverCrypt offers two advantages over accessing them directly (e.g. using <span class="caps">HACL</span>*): <em>multiplexing</em> and <em>agile interfaces</em>.</p>
<ul>
<li>
<p><strong>EverCrypt provides multiplexing:</strong></p>
<p><em>Multiplexing</em> means that EverCrypt automatically chooses the fastest available implementation of a given primitive, with no input required from the developer.</p>
<p><span class="caps">HACL</span>* may offer multiple implementations of some cryptographic algorithms.
<a href="#blake-2">In the case of <span class="caps">BLAKE2</span></a> there is a choice between</p>
<ul>
<li>a C implementation that is portable and runs on all platforms, and</li>
<li>a faster but less portable vectorized implementation that only works on some platforms.</li>
</ul>
<p>For other primitives, there might be a choice between a portable C implementation, and one that uses verified assembly.</p>
</li>
<li>
<p><strong>EverCrypt provides agile interfaces:</strong></p>
<p>EverCrypt offers interfaces that group algorithms that perfom the same general function, e.g. hashing or <span class="caps">HMAC</span>.
Users call a single function <em>hash</em> and pass the name of specific algorithm desired as a parameter.
EverCrypt will then also multiplex to choose the specific <em>implementation</em> of that algorithm.<sup id="fnref:big-deal"><a class="footnote-ref" href="#fn:big-deal">6</a></sup></p>
</li>
</ul>
<p>See <a href="https://project-everest.github.io/assets/evercrypt.pdf">this paper</a> for a much more in-depth look at how EverCrypt works and how it supports the development of verified applications.</p>
<p><a name="OCaml-API"></a></p>
<h3 id="33-the-ocaml-api"><a class="toclink" href="#33-the-ocaml-api">3.3 The OCaml <span class="caps">API</span></a></h3>
<p>So now we have a cryptographic library (<span class="caps">HACL</span>*) and a cryptographic provider (EverCrypt).</p>
<p>There remains a technical issue: calling C functions (like those of <span class="caps">HACL</span>* and EveryCrypt) from an OCaml application (like Octez).
How best to call a C function from OCaml?</p>
<p>It depends!
One common mechanism is a <strong>foreign function interface</strong> (<span class="caps">FFI</span>).
The <a href="https://caml.inria.fr/pub/docs/manual-ocaml/intfc.html">OCaml <span class="caps">FFI</span></a> allows developers to call C functions; it’s up to the developer to write a binding that matches the signature of the C function, and to manage the relevant memory.</p>
<p>Interfacing with an external library using the OCaml <span class="caps">FFI</span> works, but it can be error-prone, time-consuming, and can duplicate effort for each project that uses the external library.</p>
<p>To ameliorate this and facilitate adoption in the OCaml ecosystem, EverCrypt and <span class="caps">HACL</span>* support an OCaml <span class="caps">API</span> primarily developed by Nomadic Labs.
Released as the <a href="https://opam.ocaml.org/packages/hacl-star/"><code>hacl-star</code> package on opam</a>, this provides an idiomatic, high-level interface to the EverCrypt and <span class="caps">HACL</span>* APIs.
Starting with the recently-released version 0.4, the <span class="caps">API</span> is also <a href="https://hacl-star.github.io/ocaml_doc/hacl-star/index.html">fully documented</a>.</p>
<p>As is often the case in programming, a more <em>convenient</em> interface is also a <em>safer</em> one, and <code>hacl-star</code> offers safety benefits compared to binding with the C code directly:</p>
<ol>
<li>The lower-level bindings which interact directly with the C code are automatically generated as part of the compilation of F* code to C and are thus valid by construction with respect to the C code.</li>
<li>In the higher-level interface, all function calls check the <em>preconditions</em> that these functions have in F*. The formal guarantees of the verified code only hold if the arguments satisfy these preconditions (such as buffers being a certain length or not passing the same buffer as multiple arguments). This prevents developers from using the <span class="caps">API</span> incorrectly, and is therefore safer.</li>
</ol>
<p>The rest of this blog post dives into a description of how the tools above actually get invoked in the Octez codebase.
We are proud of our code and would love you to read this — but you are also welcome to skip to the <a href="#further-reading">further reading</a>!</p>
<h2 id="4-a-deep-dive-into-the-ocaml-api"><a class="toclink" href="#4-a-deep-dive-into-the-ocaml-api">4. A deep dive into the OCaml <span class="caps">API</span></a></h2>
<p>The OCaml <span class="caps">API</span> is split into</p>
<ul>
<li>a low-level part called <code>hacl-star-raw</code> (this contains the actual cryptography) and</li>
<li>a high-level idiomatic interface <code>hacl-star</code> (this makes the low-level part much easier to use).</li>
</ul>
<p>Let’s look in more detail at how <code>hacl-star-raw</code> replaces manually-written bindings with automatically generated ones, and how <code>hacl-star</code> builds on top of this to offer a convenient, safer <span class="caps">API</span>.</p>
<h3 id="41-the-low-level-api-hacl-star-raw"><a class="toclink" href="#41-the-low-level-api-hacl-star-raw">4.1 The low-level <span class="caps">API</span> <code>hacl-star-raw</code></a></h3>
<p>Consider the example of the <span class="caps">SHA</span>-256 function from <span class="caps">HACL</span>*.
Its C signature is:
<a name="sha2-c"></a></p>
<div class="highlight"><pre><span></span><code><span class="kt">void</span><span class="w"> </span><span class="nf">Hacl_Hash_SHA2_hash_256</span><span class="p">(</span><span class="kt">uint8_t</span><span class="w"> </span><span class="o">*</span><span class="n">input</span><span class="p">,</span><span class="w"> </span><span class="kt">uint32_t</span><span class="w"> </span><span class="n">input_len</span><span class="p">,</span><span class="w"> </span><span class="kt">uint8_t</span><span class="w"> </span><span class="o">*</span><span class="n">dst</span><span class="p">);</span>
</code></pre></div>
<p>To call it from OCaml we can write a C stub file and match the C types with compatible OCaml <span class="caps">FFI</span> types:</p>
<div class="highlight"><pre><span></span><code><span class="cp">#include</span><span class="w"> </span><span class="cpf"><caml/mlvalues.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf"><caml/bigarray.h></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf">"Hacl_Hash.h"</span>
<span class="n">CAMLprim</span><span class="w"> </span><span class="n">value</span><span class="w"> </span><span class="n">ml_Hacl_Hash_SHA2_hash_256</span><span class="p">(</span><span class="n">value</span><span class="w"> </span><span class="n">input</span><span class="p">,</span><span class="w"> </span><span class="n">value</span><span class="w"> </span><span class="n">input_len</span><span class="p">,</span><span class="w"> </span><span class="n">value</span><span class="w"> </span><span class="n">dst</span><span class="p">)</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="n">Hacl_Hash_SHA2_hash_256</span><span class="p">(</span><span class="n">String_val</span><span class="p">(</span><span class="n">input</span><span class="p">),</span>
<span class="w"> </span><span class="n">Int_val</span><span class="p">(</span><span class="n">input_len</span><span class="p">)</span>
<span class="w"> </span><span class="n">String_val</span><span class="p">(</span><span class="n">dst</span><span class="p">));</span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">Val_unit</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>Then we can bind this external function to an OCaml function which we can call in the rest of the code as we would any OCaml function:</p>
<div class="highlight"><pre><span></span><code><span class="k">external</span> <span class="n">sha2_256_hash</span> <span class="o">:</span> <span class="nn">Bytes</span><span class="p">.</span><span class="n">t</span> <span class="o">-></span> <span class="kt">int</span> <span class="o">-></span> <span class="nn">Bytes</span><span class="p">.</span><span class="n">t</span> <span class="o">-></span> <span class="kt">unit</span> <span class="o">=</span>
<span class="s2">"ml_Hacl_Hash_SHA2_hash_256"</span> <span class="o">[@@</span><span class="n">noalloc</span><span class="o">]</span>
</code></pre></div>
<p>However, there’s a simpler way to write bindings.</p>
<p>Using the <a href="https://github.com/ocamllabs/ocaml-ctypes">Ctypes library</a>, we can just write OCaml declarations for the C functions that we want to bind, and the library takes care of the rest.
A Ctypes declaration for the <span class="caps">SHA</span>-256 example above would look like this:
<a name="sha2-ctypes"></a></p>
<div class="highlight"><pre><span></span><code><span class="k">open</span> <span class="nc">Ctypes</span>
<span class="k">module</span> <span class="nc">Bindings</span><span class="o">(</span><span class="nc">F</span><span class="o">:</span><span class="nn">Cstubs</span><span class="p">.</span><span class="nc">FOREIGN</span><span class="o">)</span> <span class="o">=</span>
<span class="k">struct</span>
<span class="k">open</span> <span class="nc">F</span>
<span class="k">let</span> <span class="n">hacl_Hash_SHA2_hash_256</span> <span class="o">=</span>
<span class="n">foreign</span> <span class="s2">"Hacl_Hash_SHA2_hash_256"</span>
<span class="o">(</span><span class="n">ocaml_bytes</span> <span class="o">@-></span> <span class="n">uint32_t</span> <span class="o">@-></span> <span class="n">ocaml_bytes</span> <span class="o">@-></span> <span class="n">returning</span> <span class="n">void</span><span class="o">)</span>
<span class="k">end</span>
</code></pre></div>
<p>This is cleaner than before, but we’ve only bound a single function – the <span class="caps">SHA</span>-256 hash. We would need to write one such binding for every function from the C library that we want to use in OCaml.</p>
<p>We now take this a step further by having the KreMLin tool automatically generate these Ctypes declarations, at the same time as the C code.
This automation saves time and effort and furthermore it ensures that the signatures are correct with respect to the original F* code and the extracted C code, and that they remain in sync when the F* code changes.</p>
<p>This functionality in KreMLin can be fine-tuned,</p>
<ul>
<li>allowing users to specify which of the resulting C modules will come with Ctypes bindings, and</li>
<li>correctly inferring which dependencies also need to be bound.</li>
</ul>
<p>The output is a collection of bindings that resemble the <a href="#sha2-ctypes">snippet above</a>, along with the required Ctypes boilerplate and a <code>.depend</code> file listing dependencies between the generated bindings to be used as part of the build.</p>
<p>Thus, we generate bindings for the entirety of the EverCrypt/<span class="caps">HACL</span>* code every time a new snapshot is produced.
This is then all packaged as the <a href="https://opam.ocaml.org/packages/hacl-star-raw/"><code>hacl-star-raw</code></a> opam package.</p>
<h3 id="42-the-high-level-api-hacl-star"><a class="toclink" href="#42-the-high-level-api-hacl-star">4.2 The high-level <span class="caps">API</span> <code>hacl-star</code></a></h3>
<p><code>hacl-star-raw</code> is better than writing C bindings by hand, but it still offers a low-level, C style interface to the library.</p>
<p>To complement this we developed a handwritten idiomatic OCaml <span class="caps">API</span>.
This further improves convenience and safety; primarily because the preconditions of the original F* functions (which are lost when compiling the F* code to C) can be enforced at runtime.</p>
<p>To illustrate, let’s look at the original <a href="https://github.com/project-everest/hacl-star/blob/927a70b03df989356ab449dad579d949fce8d116/code/hash/Hacl.Hash.Definitions.fst#L229">F* signature</a> (<a href="https://archive.softwareheritage.org/swh:1:cnt:0248150822f8545db8f6c92a732243efc7944f49;origin=https://github.com/project-everest/hacl-star;visit=swh:1:snp:a31eaf516010b41b28c1f090f9e4bb873440933a;anchor=swh:1:rev:f95b54abf9076a7a072a54750b40dc9f85039b74;path=/code/hash/Hacl.Hash.Definitions.fst;lines=229">permalink</a>) of the <span class="caps">SHA</span>-256 function above:</p>
<div class="highlight"><pre><span></span><code><span class="k">module</span> <span class="nc">B</span> <span class="o">=</span> <span class="nn">LowStar</span><span class="p">.</span><span class="nc">Buffer</span>
<span class="k">let</span> <span class="n">hash_st</span> <span class="o">(</span><span class="n">a</span><span class="o">:</span> <span class="n">hash_alg</span><span class="o">)</span> <span class="o">=</span>
<span class="n">input</span><span class="o">:</span><span class="nn">B</span><span class="p">.</span><span class="n">buffer</span> <span class="n">uint8</span> <span class="o">-></span>
<span class="n">input_len</span><span class="o">:</span><span class="n">size_t</span> <span class="o">{</span> <span class="nn">B</span><span class="p">.</span><span class="n">length</span> <span class="n">input</span> <span class="o">=</span> <span class="n">v</span> <span class="n">input_len</span> <span class="o">}</span> <span class="o">-></span>
<span class="n">dst</span><span class="o">:</span><span class="n">hash_t</span> <span class="n">a</span><span class="o">-></span>
<span class="nn">ST</span><span class="p">.</span><span class="nc">Stack</span> <span class="kt">unit</span>
<span class="o">(</span><span class="n">requires</span> <span class="o">(</span><span class="k">fun</span> <span class="n">h</span> <span class="o">-></span>
<span class="nn">B</span><span class="p">.</span><span class="n">live</span> <span class="n">h</span> <span class="n">input</span> <span class="o">/</span><span class="err">\</span>
<span class="nn">B</span><span class="p">.</span><span class="n">live</span> <span class="n">h</span> <span class="n">dst</span> <span class="o">/</span><span class="err">\</span>
<span class="nn">B</span><span class="p">.</span><span class="n">disjoint</span> <span class="n">input</span> <span class="n">dst</span> <span class="o">/</span><span class="err">\</span>
<span class="nn">B</span><span class="p">.</span><span class="n">length</span> <span class="n">input</span> <span class="o"><=</span> <span class="n">max_input_length</span> <span class="n">a</span><span class="o">))</span>
<span class="o">(</span><span class="n">ensures</span> <span class="o">(</span><span class="k">fun</span> <span class="n">h0</span> <span class="o">_</span> <span class="n">h1</span> <span class="o">-></span>
<span class="nn">B</span><span class="p">.</span><span class="o">(</span><span class="n">modifies</span> <span class="o">(</span><span class="n">loc_buffer</span> <span class="n">dst</span><span class="o">)</span> <span class="n">h0</span> <span class="n">h1</span><span class="o">)</span> <span class="o">/</span><span class="err">\</span>
<span class="nn">Seq</span><span class="p">.</span><span class="n">equal</span> <span class="o">(</span><span class="nn">B</span><span class="p">.</span><span class="n">as_seq</span> <span class="n">h1</span> <span class="n">dst</span><span class="o">)</span> <span class="o">(</span><span class="nn">Spec</span><span class="p">.</span><span class="nn">Agile</span><span class="p">.</span><span class="nn">Hash</span><span class="p">.</span><span class="n">hash</span> <span class="n">a</span> <span class="o">(</span><span class="nn">B</span><span class="p">.</span><span class="n">as_seq</span> <span class="n">h0</span> <span class="n">input</span><span class="o">))))</span>
<span class="o">[...]</span>
<span class="k">val</span> <span class="n">hash_256</span><span class="o">:</span> <span class="n">hash_st</span> <span class="nc">SHA2_256</span>
</code></pre></div>
<p>Let’s break <code>hash_st</code> down:</p>
<ol>
<li>We can see that <code>hash_256</code> is an instantiation of a generic <span class="caps">SHA</span>-2 function <code>sha_st</code>, parameterized by the specific algorithm. This is a pervasive pattern throughout the library.</li>
<li>
<p>Just as we specified in the OCaml binding above, it takes three arguments:</p>
<ol>
<li><code>input</code>, which is a <code>uint8_t</code> buffer,</li>
<li><code>input_len</code>, with a <a href="https://www.fstar-lang.org/tutorial/tutorial.html#sec-refinement-types"><em>refinement</em></a> specifying that the size of <code>input</code> must be equal to <code>input_len</code>, and</li>
<li><code>dst</code> of type <code>hash_t a</code>, which is also a <code>uint8_t</code> buffer of the correct digest size for algorithm <code>a</code>.</li>
</ol>
</li>
</ol>
<p>Glossing over some of the details, we see in the <code>requires</code> clause above that the function <code>hash_st</code> (and therefore <code>hash_256</code> too) has certain liveness, disjointness, and other preconditions. In particular:</p>
<ul>
<li>the input buffer must be smaller than the maximum allowed size for the specific algorithm (<code>B.length input <= max_input_length a</code>), and</li>
<li><code>hash_256</code> is guaranteed to modify only <code>dst</code>, and <code>dst</code> will contain the result of the hash as defined in the spec.</li>
</ul>
<p>In the original F* these preconditions are statically checked when compiling code that calls <code>hash_256</code>.
But in OCaml we only get to work with the <em>extracted C code</em>, in which this information has been erased (as seen in the <a href="#sha2-c">C signature</a> of the function).
Users of the C library <em>must</em> make sure that the arguments they pass respect the original preconditions of these functions.</p>
<p>In the OCaml <span class="caps">API</span>, we do check all of these preconditions at runtime. For example, this is what the functor used internally for hash functions roughly looks like:</p>
<div class="highlight"><pre><span></span><code><span class="k">module</span> <span class="nc">Make_HashFunction</span> <span class="o">(</span><span class="nc">C</span><span class="o">:</span> <span class="nc">Buffer</span><span class="o">)</span>
<span class="o">(</span><span class="nc">Impl</span> <span class="o">:</span> <span class="k">sig</span>
<span class="k">val</span> <span class="n">hash_alg</span> <span class="o">:</span> <span class="n">alg</span>
<span class="k">val</span> <span class="n">hash</span> <span class="o">:</span> <span class="nn">C</span><span class="p">.</span><span class="n">buf</span> <span class="o">-></span> <span class="n">uint32</span> <span class="o">-></span> <span class="nn">C</span><span class="p">.</span><span class="n">buf</span> <span class="o">-></span> <span class="kt">unit</span>
<span class="k">end</span><span class="o">)</span>
<span class="o">=</span> <span class="k">struct</span>
<span class="k">let</span> <span class="n">hash</span> <span class="o">~</span><span class="n">msg</span> <span class="o">~</span><span class="n">digest</span> <span class="o">=</span>
<span class="n">check_max_input_len</span> <span class="nn">Impl</span><span class="p">.</span><span class="n">hash_alg</span> <span class="o">(</span><span class="nn">C</span><span class="p">.</span><span class="n">size</span> <span class="n">msg</span><span class="o">);</span>
<span class="k">assert</span> <span class="o">(</span><span class="nn">C</span><span class="p">.</span><span class="n">size</span> <span class="n">digest</span> <span class="o">=</span> <span class="n">digest_len</span> <span class="nn">Impl</span><span class="p">.</span><span class="n">hash_alg</span><span class="o">);</span>
<span class="k">assert</span> <span class="o">(</span><span class="nn">C</span><span class="p">.</span><span class="n">disjoint</span> <span class="n">msg</span> <span class="n">digest</span><span class="o">);</span>
<span class="nn">Impl</span><span class="p">.</span><span class="n">hash</span> <span class="o">(</span><span class="nn">C</span><span class="p">.</span><span class="n">ctypes_buf</span> <span class="n">msg</span><span class="o">)</span> <span class="o">(</span><span class="nn">C</span><span class="p">.</span><span class="n">size_uint32</span> <span class="n">msg</span><span class="o">)</span> <span class="o">(</span><span class="nn">C</span><span class="p">.</span><span class="n">ctypes_buf</span> <span class="n">digest</span><span class="o">)</span>
<span class="k">end</span>
</code></pre></div>
<p><code>Make_HashFunction</code> is parameterized both by</p>
<ul>
<li><code>C</code>, which is the data type we want to use to represent C buffers (we currently use <code>Bytes</code>, but <code>Bigstring</code> is also possible), and</li>
<li><code>Impl</code>, the specific hash function implementation.</li>
</ul>
<p>Before calling the bound C function, the buffers are checked to have the correct size and to be disjoint (which for <code>Bytes</code> simply means checking for inequality).</p>
<p><a name="Using"></a></p>
<h3 id="43-using-the-high-level-api"><a class="toclink" href="#43-using-the-high-level-api">4.3 Using the high-level <span class="caps">API</span></a></h3>
<p>We’ve seen how the OCaml library comes together, from the <span class="caps">HACL</span>* C code, to the low-level Ctypes bindings, to the idiomatic OCaml <span class="caps">API</span>.
Now let’s look at how it can be used.</p>
<p>Most cryptographic algorithms exposed through this OCaml <span class="caps">API</span> can be called in more than one way, to suit different use cases.
Our <span class="caps">SHA</span>-256 example above will conveniently illustrate them.</p>
<p>The <span class="caps">API</span> is split in two interfaces:</p>
<ul>
<li><a href="https://hacl-star.github.io/ocaml_doc/hacl-star/Hacl_star/Hacl/index.html"><code>Hacl</code></a>, which directly exposes the portable C implementations</li>
<li><a href="https://hacl-star.github.io/ocaml_doc/hacl-star/Hacl_star/EverCrypt/index.html"><code>EverCrypt</code></a>, which exposes the agile and/or multiplexing interfaces</li>
</ul>
<p>In <code>Hacl</code>, <span class="caps">SHA</span>-256 can be used in two styles:</p>
<ol>
<li><a href="https://hacl-star.github.io/ocaml_doc/hacl-star/Hacl_star/Hacl/SHA2_256/index.html#val-hash"><code>Hacl.SHA2_256.hash</code></a> which takes a buffer representing the message and returns the digest</li>
<li><a href="https://hacl-star.github.io/ocaml_doc/hacl-star/Hacl_star/Hacl/SHA2_256/Noalloc/index.html#val-hash"><code>Hacl.SHA2_256.Noalloc.hash</code></a> which, more in keeping with the C style, takes as inputs both the message buffer and the output buffer into which the digest will be written</li>
</ol>
<p>The first style is usually more convenient, but the second style can be useful in cases where an output buffer has already been allocated and we don’t want to allocate a new one.
The choice is the programmer’s to make: most modules in <code>Hacl</code> and <code>EverCrypt</code> offer both styles above.</p>
<p><code>EverCrypt</code> offers three further options:</p>
<ol>
<li><a href="https://hacl-star.github.io/ocaml_doc/hacl-star/Hacl_star/EverCrypt/SHA2_256/index.html"><code>EverCrypt.SHA2_256</code></a> is an identical interface to <code>Hacl.SHA2_256</code>, but with a different underlying implementation: <code>Hacl.SHA2_256</code> uses the portable C implementation of <span class="caps">SHA</span>-256; whereas <code>EverCrypt.SHA2_256</code> uses the multiplexing EverCrypt interface which automatically uses code relying on <a href="https://en.wikipedia.org/wiki/Intel_SHA_extensions">Intel <span class="caps">SHA</span> extensions</a> if the architecture supports it.</li>
<li>
<p><a href="https://hacl-star.github.io/ocaml_doc/hacl-star/Hacl_star/EverCrypt/Hash/index.html#val-hash"><code>EverCrypt.Hash.hash</code></a> is an agile and multiplexing interface to all hashing functions supported in EverCrypt and it is parameterized by the hashing algorithm:
<pre><code>
let digest = EverCrypt.Hash.hash ~alg:SHA2_256 ~msg
</code></pre></p>
</li>
<li>
<p>The <a href="https://hacl-star.github.io/ocaml_doc/hacl-star/Hacl_star/EverCrypt/Hash/index.html#streaming-interface"><code>EverCrypt.Hash</code> incremental hashing interface</a> can be used when we need to update the internal state repeatedly before generating a digest:
<pre><code>
let st = EverCrypt.Hash.init ~alg:SHA2_256 in
EverCrypt.Hash.update ~st ~msg; (* can be called multiple times *)
let digest = EverCrypt.Hash.finish ~st
</code></pre></p>
</li>
</ol>
<p><a id="further-reading"></a></p>
<h2 id="5-further-reading"><a class="toclink" href="#5-further-reading">5. Further reading</a></h2>
<p>Taken together, these changes constitute a significant investment in improving our codebase, which has brought us up-to-date with the best technology available and allowed us to increase reliability and flexibility, without compromising on performance.</p>
<p>You can read further in:</p>
<ul>
<li>The <a href="https://hacl-star.github.io/ocaml_doc/hacl-star/index.html">OCaml <span class="caps">API</span> documentation</a>.</li>
<li>For web development, an official <a href="https://hacl-star.github.io/Obtaining.html#javascript">JavaScript <span class="caps">API</span></a> is available, exposing the version of <span class="caps">HACL</span>* extracted to WebAssembly.</li>
<li>For more examples using other algorithms, see the <a href="https://github.com/project-everest/hacl-star/tree/master/bindings/ocaml/tests">unit tests</a>.</li>
<li><a href="https://hacl-star.github.io/HaclValeEverCrypt.html">EverCrypt</a> is free and open-source, for you to use.</li>
<li>See <a href="https://project-everest.github.io/assets/evercrypt.pdf">this paper</a> for a more in-depth look at how EverCrypt works and how it supports the development of verified applications.</li>
</ul>
<p>We hope this survey has been informative and may inspire you to use EverCrypt in your own OCaml projects too!</p>
<h2 id="acknowledgements"><a class="toclink" href="#acknowledgements">Acknowledgements</a></h2>
<p>Thanks to Karthikeyan Bhargavan, Natalia Kulatova, and Marina Polubelova of the <a href="https://prosecco.gforge.inria.fr/">Prosecco team at <span class="caps">INRIA</span></a>, and to <a href="https://www.microsoft.com/en-us/research/people/protz/">Jonathan Protzenko</a> of Microsoft Research, for their help, support, and collaboration.
At Nomadic Labs, the work reported on in this article was carried out principally by Victor Dumitrescu.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:0">
<p><span class="caps">HACL</span>* was developed as part of <a href="https://project-everest.github.io/">Project Everest</a>, a collaboration between Inria Paris, Microsoft Research, and other institutions and contributors. <a class="footnote-backref" href="#fnref:0" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>secp256k1 (Bitcoin’s signature scheme) is the only core primitive that is not currently implemented in the <span class="caps">HACL</span>* library; we just import the code directly from Bitcoin. <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 2 in the text">↩</a><a class="footnote-backref" href="#fnref2:3" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:2bornot2b">
<p><span class="caps">BLAKE2</span> comes in two flavours: <em>BLAKE2b</em> optimized for 64-bit platforms and produces digests between 1 and 64 bytes long, and <em>BLAKE2s</em> optimized for 32-bit platforms and produces digests between 1 and 32 bytes.
There is also 4-way parallel BLAKE2bp and 8-way parallel BLAKE2sp, as well as BLAKE2x which can produce digests of arbitrary length.
None of that matters for this article, except to note that the Tezos Octez implementation uses BLAKE2b and for the purposes of this article, BLAKE2b = <span class="caps">BLAKE2</span>. <a class="footnote-backref" href="#fnref:2bornot2b" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:4">
<p>See <a href="https://eprint.iacr.org/2020/572">Section 4.1 of the paper</a> <em>“On Intel processors, vectorization speeds up Blake2 by about 30%”</em>.</p>
<p>In practice the computational cost of cryptographic operations in Tezos is small compared to that of the other components of the network, so the discussion of performance just confirms that our tooling upgrade is (slightly faster and certainly) not slower than the version it replaces.
We checked this empirically with a benchmark, running a Tezos node for a fixed (large) set of operations with different <span class="caps">BLAKE2</span> implementations. There was no noticeable difference in performance, even using <span class="caps">BLAKE2</span> implementations with significantly different performance when directly compared. <a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:caveat">
<p>Not all primitives in <span class="caps">HACL</span>* are accessible through EverCrypt, and we don’t use all of them through EverCrypt. Primitives that only have a portable C implementation and are not part of some agile interface, don’t benefit from being included in EverCrypt. Conversely, all the primitives in EverCrypt are available in <span class="caps">HACL</span>*, but parts of their implementations can sometimes come from a different source, e.g. Vale. <a class="footnote-backref" href="#fnref:caveat" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:big-deal">
<p>Agile Interfaces are just a limited form of higher-order programming (meaning functions passed as parameters to other functions), but remember that this is accessed as a C library. Passing functions as parameters is a bigger deal in the C world than e.g. in the OCaml world. <a class="footnote-backref" href="#fnref:big-deal" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
</ol>
</div>Timelock: a solution to miner/block producer extractable value2021-10-11T12:00:00+02:002021-10-11T12:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-10-11:/timelock-a-solution-to-minerblock-producer-extractable-value.html<p>We solve a block-producer arbitrage issue with a new Timelock primitive, which can be thought of as “commitment + forced reveal after time T”</p><p>Block Producer Extractable Value (<span class="caps">BPEV</span>) — called <span class="caps">MEV</span> on proof-of-work chains (Miner Extractable Value) — is a form of arbitrage present in most decentralized finance applications on blockchains. It is undesirable, because it imposes costs on users, to the benefit of block producers (examples below). </p>
<p>To help improve this situation we propose in <a href="https://forum.tezosagora.org/t/announcing-tezos-8th-protocol-upgrade-proposal-hangzhou/3695">Hangzhou</a> a new Michelson smart contract instruction, based on a cryptographic primitive named <strong>Timelock</strong> (<a href="https://tezos.gitlab.io/alpha/timelock.html">technical documentation</a>). The solution we propose allows to temporarily hide the payload of a transaction sent to a smart contract for a period of time greater than the time it takes to include the transaction in a block.</p>
<p>In this way there is no leakage of information at the <span class="caps">P2P</span> level and no reordering of transactions can be applied in order to arbitrage the smart contract.</p>
<p>We argue that this tackles the issue of <span class="caps">BPEV</span> at its core.</p>
<p>In this post we explain <span class="caps">BPEV</span> and its relationship to the fees market.</p>
<p>We identify the core problem as a leakage of information between different layers of the architecture of modern blockchains. We argue that this problem is not inherent to blockchains and thus can be solved with the right cryptographic tool.</p>
<p>We explain the advantages of cryptographic commitments and why they are not enough to tackle our problem.</p>
<p>Finally we show how the Timelock primitive improves over commitments and we sketch how to use it to protect a smart contract from <span class="caps">BPEV</span>.</p>
<h1 id="block-producer-extractable-value-bpev"><a class="toclink" href="#block-producer-extractable-value-bpev">Block Producer Extractable Value (<span class="caps">BPEV</span>)</a></h1>
<h2 id="block-producers-and-fees-market"><a class="toclink" href="#block-producers-and-fees-market">Block Producers and fees market</a></h2>
<p>To understand <span class="caps">BPEV</span> it is useful to recall the role of block producers in a blockchain.</p>
<p>To enjoy the benefits of decentralization, blockchains have a permissionless design so as to attract a large number of <strong>block producers</strong>.<sup id="fnref:not-all"><a class="footnote-ref" href="#fn:not-all">1</a></sup></p>
<p>In Tezos for example anybody with a computer and the necessary stake can become one. Today the network is run by around 400 block producers. In order to motivate block producers to operate on the network they obtain an economic incentive for each block produced.</p>
<p>This incentive is usually composed of a fixed amount created from inflation (block reward) and, less importantly, the fees of all transactions included in the block.</p>
<p>Because each block can only include a finite amount of transactions, a fee market forms where users bid for inclusion of their transactions. The block producer naturally prioritizes transactions with a higher fee.</p>
<p>This system popularized by Bitcoin works well for simple transfers. However things get more complicated when transactions are intended for complicated smart contracts, and can for example affect the price of an asset when executed. A first hint of the problem is visible from the fact that the order in which transactions are included in a block is the prerogative of the block producer and is outside of the control of the transaction emitter.</p>
<h2 id="what-is-bpev"><a class="toclink" href="#what-is-bpev">What is <span class="caps">BPEV</span>?</a></h2>
<p>A large class of smart contracts (typically ones related to DeFi) are sensitive to the ordering of transactions. By scanning the transactions on the network, arbitrageurs can exploit the fees market to manipulate smart contracts and produce undesirable arbitrages.</p>
<p>We can see a simple example in which a smart contract automatically computes the price of an asset based on supply and demand (e.g. an Automated Market Maker). A user wishing to buy this asset sends a transaction with a fee. An arbitrageur monitoring the <span class="caps">P2P</span> network then produces a transaction to buy the same asset but with a higher fee, so as to have priority in the next block. When the arbitrageur’s transaction is executed, the price of the asset is adjusted by the smart contract and the honest transaction buys the asset at a higher price from the arbitrageur, who then immediately sells the asset and makes a profit, a so-called “sandwich”.</p>
<p>This simple example can be reminiscent of front-running in a classic financial market, although a key difference is that front-running in financial market involves the malicious use of client’s information by fiduciaries while, in this scenario, no direct breach of trust is involved.<sup id="fnref:name"><a class="footnote-ref" href="#fn:name">2</a></sup></p>
<p>The phenomenom is particularly relevant on blockchains since the effect of a block can be determined algorithmically, and the process can be easily automated and leads to more sophisticated arbitrages that are used in practice.</p>
<p>It is important to note that several actors can compete among themselves for priority in a block thus causing spikes in the fee prices and reducing their margin of gain.</p>
<p>In the end, block producers are extracting the most value from this practice because of the inherent power of their role.</p>
<p>The value in this attack is therefore called <strong>Miner Extractable Value</strong> (<span class="caps">MEV</span>) on Ethereum.
Tezos faces a similar phenomenom, that we call <strong>Block Producer Extractable Value</strong> to extend the terminology to proof-of-stake networks (<span class="caps">BPEV</span>).</p>
<p>One concrete account of <span class="caps">MEV</span> (on the Ethereum network) is described in the article <a href="https://web.archive.org/web/20210815101318/https://www.paradigm.xyz/2020/08/ethereum-is-a-dark-forest/">“Ethereum is a dark forest”</a>.
The author suffered a 12,000 <span class="caps">USD</span> loss on a transaction which was submitted and arbitraged away.
See also</p>
<ul>
<li><a href="https://docs.flashbots.net/">Flashbots docs</a> (<a href="https://web.archive.org/web/20210524232632/https://docs.flashbots.net/">permalink</a>), and</li>
<li>a detailed paper <a href="https://arxiv.org/pdf/1904.05234.pdf">“Flash Boys 2.0: Frontrunning, Transaction Reordering, and Consensus Instability in Decentralized Exchanges”</a>.</li>
</ul>
<p>which shows the extent of the problem.</p>
<h2 id="the-core-of-the-problem"><a class="toclink" href="#the-core-of-the-problem">The core of the problem</a></h2>
<p>It is important to understand that this is an architectural problem. Blockchains are usually separated into three layers: a <span class="caps">P2P</span> (peer-to-peer) layer, a consensus layer, and a smart contracts layer.
Let’s take an example to see the life of a transaction throughout the three layers in a typical blockchain.
A user creates a transaction of the form (source, fees, payload).
The source is the address of the user, the fees are meant for the block producer, and the payload is intended for a smart contract.
The user then broadcasts the transaction in the clear on a gossip <span class="caps">P2P</span> network.
The consensus layer validates the transaction and includes it in a block.
In principle the validation could only ensure that the source has enough balance to pay the fees.
Once the transaction is included in a block by the consensus layer, the recipient smart contract executes the payload.</p>
<p>Note however that the <span class="caps">P2P</span> and consensus layers have access to the payload even if it’s not necessary for their operation.
This unnecessary leakage of information is the core cause of <span class="caps">BPEV</span>.</p>
<p>As we will explain in the rest of this blogpost our solution consists of hiding the payload from the <span class="caps">P2P</span> layer.
Note that this is a non-trivial task as a smart contract cannot hold secret keys.</p>
<p>Users of the Tezos network might recall the <a href="https://tezos.gitlab.io/active/sapling.html">integration of the Sapling protocol</a> in the Edo amendment and wonder if it could solve the problem.
Indeed in some cases Sapling can be used to alleviate <span class="caps">BPEV</span>, however it offers a very strong notion of privacy which in many cases is not required and which may limit the expressivity of the smart contract.</p>
<p>The solution we propose, called <strong>Timelock</strong>, allows to temporarily hide the payload of a smart contract for the time necessary to have it included in a block.
The payload is then decrypted and the smart contract can operate as before.</p>
<h1 id="cryptography-and-fairness"><a class="toclink" href="#cryptography-and-fairness">Cryptography and fairness</a></h1>
<p>Before explaining our solution based on Timelock, we digress to explain the concept of fairness through the children’s game rock-paper-scissor.</p>
<h2 id="the-commit-and-reveal-technique"><a class="toclink" href="#the-commit-and-reveal-technique">The commit-and-reveal technique</a></h2>
<p>Consider the game of <a href="https://en.wikipedia.org/wiki/Rock_paper_scissors">rock-paper-scissors</a>.
In this game, it is key that players reveal their moves simultaneously — or at least within a time interval less than a human reaction time.
Unfortunately if we play the game on a blockchain, computers have ample time to react to broadcasted transactions on the <span class="caps">P2P</span> layer before they are included.
This makes it impossible for our children to play a naïve version of rock-paper-scissors across a blockchain.</p>
<h2 id="how-to-play-rock-paper-scissors-over-a-blockchain-using-commitments"><a class="toclink" href="#how-to-play-rock-paper-scissors-over-a-blockchain-using-commitments">How to play rock-paper-scissors over a blockchain, using commitments</a></h2>
<p>Using a <strong>commitment scheme</strong>, our children could still in principle play rock-paper-scissors on our blockchain.
A commitment scheme is a well-known cryptographic primitive in which we commit to having <em>made</em> a decision, without immediately revealing what that decision is.</p>
<p>Formally, we require two properties:</p>
<ul>
<li><strong>Hiding property:</strong> given a value I can commit to this value. However, my commitment gives no information about the value committed to.</li>
<li><strong>Binding property:</strong> If I reveal my value, this is guaranteed to be equal to the value that I committed to earlier.</li>
</ul>
<p>We can now devise a protocol to play our game:</p>
<ul>
<li>Player 1 chooses a move <span class="math">\(M_1\in\{rock, paper, scissors\}\)</span> and broadcasts a commitment <span class="math">\(C(M_1)\)</span> of <span class="math">\(M_1\)</span>.</li>
<li>Player 2 receives <span class="math">\(C_1\)</span> and sends a move <span class="math">\(M_2\in\{rock,paper,scissors\}\)</span> in the clear.</li>
<li>Player 1 receives <span class="math">\(M_2\)</span> and reveals <span class="math">\(M_1\)</span>.</li>
</ul>
<p>The hiding property gives Player 1 confidence that Player 2 cannot use the value of <span class="math">\(M_1\)</span> to choose <span class="math">\(M_2\)</span>, and the binding property gives Player 2 confidence that <span class="math">\(M_1\)</span> was indeed the chosen move.
Thanks to commitments, both players are protected, neither can gain an advantage, and the game is fair.</p>
<h2 id="commitments-and-bpev"><a class="toclink" href="#commitments-and-bpev">Commitments and <span class="caps">BPEV</span></a></h2>
<p>Commitments are a partial solution to <span class="caps">BPEV</span>.</p>
<p>Users can <em>commit</em> to transactions, and block producers can include these in the blockchain.
However, block producers are kept honest in the sense that the hiding property means that they cannot interrogate the commitments for information with which to extract value for example by reordering or inserting transactions.
The binding property means that commitments really <em>do</em> commit users to the transactions that they chose.</p>
<p>The overall <span class="caps">P2P</span>-Consensus-SmartContracts blockchain architecture, is untouched.
Later — just as in the game above — users can reveal their actual transactions.</p>
<p>However, there is a catch: the binding property guarantees that a user cannot reveal a <em>different</em> transaction, but does not force the user to <em>reveal</em>.
Users could spam the system with transactions and only reveal the ones that suit them.<sup id="fnref:rock"><a class="footnote-ref" href="#fn:rock">3</a></sup></p>
<p>So commitments are a step in a good direction, but they do not fully solve the problem.</p>
<h1 id="timelock-and-bpev"><a class="toclink" href="#timelock-and-bpev">Timelock and <span class="caps">BPEV</span></a></h1>
<p>We propose a solution to alleviate this issue which is relatively easy to implement and is based on an old cryptographic technique called <strong>time-lock encryption</strong> (see <a href="http://www.hashcash.org/papers/time-lock.pdf"><em>Time-lock puzzles and timed-release Crypto</em></a> for more details).</p>
<p>Time lock encryption allows to encrypt a message such that it can be decrypted in two ways:
it can be decrypted rapidly using the author’s secret;
or a long and time-consuming computation can decrypt the ciphertext without requiring access to the author’s secret.
The duration of this computation can be set to an arbitrary predetermined constant <span class="math">\(T\)</span>.
In addition, a proof of the correctness of the decryption can also be produced and checked rapidly (<span class="math">\(\log T\)</span> in our case).</p>
<p>We can see intuitively that this solves the problem we had with the commitment. Indeed even if a user refuses to reveal the content of their transaction, someone else can go the slow way to decrypt it.</p>
<h2 id="general-principle"><a class="toclink" href="#general-principle">General principle</a></h2>
<p>Armed with our new Timelock cryptographic primitive we can image a smart contract implementing the following pattern:</p>
<ol>
<li>In a first period, a contract collects user-submitted and Timelock encrypted Michelson values along with some valuable deposit, such as tez.</li>
<li>In a second period, after the values are collected, the contract collects from users a decryption of the value they submitted alongside with a proof that the decryption is correct.</li>
<li>In a third period, if any value remains undecrypted, anyone can claim part of the deposit by submitting a decryption of the value, with the other part of the deposit being burnt. Different penalties can be assessed depending on whether the user merely failed to submit a decryption for their value, or if they also intentionally encrypted invalid data. Different rewards can be distributed for submitting a correct decryption. The third period needs to be long enough so that people have enough time to perform the Timelock decryption.</li>
<li>Finally, the contract can compute some function of all the decrypted data.</li>
</ol>
<p>There is generally no incentive for users not to provide the decryption of their data and thus the third period generally does not need
to take place.
However, the second period needs to be long enough so that bakers cannot easily censor submission of the decryption in a bid to later claim the reward.
Burning a part of the deposit also limits <em>grieving attacks</em>, meaning where a user gets back their whole deposit by providing the decryption, but in a way that delays everyone else.</p>
<p>As we can see our solution is generic in the way it solves the leakage issue in the <span class="caps">P2P</span> network. However it has
to be included in the design phase of an application, and many parameters need to be carefully evaluated depending on the application (incentive, latency…).</p>
<p>We propose in Hangzhou the introduction in Michelson of an opcode (<code>OPEN_CHEST</code>) and two types (<code>chest</code> and <code>chest_key</code>) allowing Timelock-encrypted values to be used inside a Michelson contract.
See more information in the <a href="https://tezos.gitlab.io/alpha/timelock.html">technical documentation</a>.</p>
<h1 id="caveats"><a class="toclink" href="#caveats">Caveats</a></h1>
<p>Although Timelock solves the core of the problem of <span class="caps">BPEV</span>, it also moves part of the complexity of validating an operation to the smart contract.
In the traditional setting, a number of errors can be detected during validation by the block producer, which can for example mark the transaction as invalid (all the while retaining the fee).
Once this information is encrypted this is no longer possible. Special cases such as decryption and deserialization errors (see the <a href="https://tezos.gitlab.io/alpha/timelock.html">technical documentation</a> for more details) are inherently application-specific issues which we therefore leave to the designer of the specific smart contract application to address. </p>
<h1 id="other-uses"><a class="toclink" href="#other-uses">Other uses</a></h1>
<p>While the problem we wished to solve was <span class="caps">BPEV</span>, Timelock might also be useful for other applications, for instance this <a href="https://eprint.iacr.org/2020/513.pdf">self-tallying fully-decentralized voting system</a>.
Intuitively, as we outline in this blog post, Timelock allows to bring some notion of fairness in interactive protocols with respect to the ordering of messages. We hope other creative uses can be made of the instruction we propose.</p>
<h1 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h1>
<p>Thanks to the introduction of Timelock in the Michelson smart contract language, transactions can hide sensitive information at the <span class="caps">P2P</span> layer, preventing any malicious reaction before the consensus layer orders them.
We believe our solution tackles the <span class="caps">BPEV</span> problem at its core, at the price of additional complexity during smart contract design.
We hope Timelock will help protect the next generation of DeFi applications in the Tezos ecosystem from the threat of <span class="caps">BPEV</span>.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:not-all">
<p>To be fair, not all blockchains are permissionless: a <em>permissioned</em> blockchain could address <span class="caps">MEV</span>/<span class="caps">BPEV</span> by fiat, by only admitting “good” block producers. We would expect this to be just as effective as (for example) the traditional global financial network is effective at ensuring honest behaviour by licensing only “good” brokers.</p>
<p>Setting aside how realistic honesty-by-fiat can be in general, for Tezos it begs the question, by replacing one problem (“How to ensure brokers stay honest”) with two (“How to ensure the permissioning authority remains honest” <em>and</em> “How to ensure brokers stay honest”). We in Tezos believe in inclusivity and decentralization <em>by design</em>, so our brief is to design a blockchain that is robust against <span class="caps">MEV</span>/<span class="caps">BPEV</span> <em>without</em> requiring a fiat permissioning authority. We believe that this is possible, feasible, and furthermore that this is the correct engineering solution; whence the topic of this blog post. <a class="footnote-backref" href="#fnref:not-all" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:name">
<p><span class="caps">MEV</span>/<span class="caps">BPEV</span> is also called <em>generalized front running</em>. We deprecate this terminology since it might suggest to the reader that there is a fiduciary relationship between block producers and transaction emitters when, in fact, none exists unless explicitly contracted into. <a class="footnote-backref" href="#fnref:name" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:rock">
<p>Imagine playing rock-paper-scissors with a three-armed alien child using commitments: it just puts down rock, paper, <em>and</em> scissors, one with each hand, then reveals whichever hand wins. <a class="footnote-backref" href="#fnref:rock" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Announcing new internship subjects for 2021!2021-09-27T16:00:00+02:002021-09-27T16:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-09-27:/announcing-new-intership-subjects-for-2021.html<p>Announcing internship subjects for the coming year</p><p><em>Are you a student in Computer Science or Cryptography? Are you excited about
tackling complex programming problems, eager to acquire a deeper knowledge in
your field of interest, and happy to work with talented and skilled staff to
build a free, open-source, decentralized blockchain architecture dedicated to
social good?</em></p>
<p>Then the Nomadic Labs internship program might be for you!</p>
<p>We at Nomadic Labs are one of the premier research and development centers of the Tezos ecosystem.
We work on the core development, evolution, and adoption of this self-amendable blockchain protocol in France, Luxembourg, Belgium, and elsewhere.
More than fifty talented engineers work with us, mixing industrial and academic skills and applying their expertise in <strong>distributed</strong>, <strong>decentralized</strong> and <strong>formally verified</strong> software.</p>
<p>In the previous internship program eight interns spent between three and six months working with us on various subjects, from
<a href="https://research-development.nomadic-labs.com/three-questions-to-nomadic-labs-interns-julien-coolen.html">peer to peer</a>
to <a href="https://research-development.nomadic-labs.com/three-questions-to-nomadic-labs-interns-valentin-chaboche.html">building testing tools</a>.
They came from Université de Paris, the École normale supérieure, the École Polytechnique — and even from the Universidad Nacional de Rosario in Argentina.</p>
<p>By joining one of our teams as an intern, you will grow your skills in a
collegiate and collaborative space, and help to build a free and open-source decentralized
ecosystem that is as committed to doing social good, as it is to technical excellence.</p>
<h1 id="internship-subjects"><a class="toclink" href="#internship-subjects">Internship subjects</a></h1>
<p>We are offering the following list of 13 internship topics — or you can come to us to propose your own (<a href="#own">see below</a>).</p>
<ul>
<li>
<p><a href="https://www.nomadic-labs.com/careers/cartography-monitoring-and-analysis-of-the-p2p-network/">Cartography, monitoring and analysis of the p2p
network</a>:
build and use cartographer nodes to analyze network metrics and topology.</p>
</li>
<li>
<p><a href="https://www.nomadic-labs.com/careers/ad-hoc-static-analysis-of-octez/">Ad-hoc Static Analysis of Octez</a>: extend or develop a pragmatic tool to analyze
octez codebase and provide metrics to improve code quality.</p>
</li>
<li>
<p><a href="https://www.nomadic-labs.com/careers/memory-footprint-analysis-of-ocaml-concurrent-programs/">Memory footprint analysis of Ocaml concurrent
programs</a>:
profile node by to evaluate the overhead of the currently used promise library</p>
</li>
<li>
<p><a href="https://www.nomadic-labs.com/careers/fat-cat-formal-acceptance-testing-of-contracts-for-administering-tokens/"><span class="caps">FAT</span> <span class="caps">CAT</span>, Formal Acceptance Testing of Contracts for Administering
Tokens</a>:
use Coq proof assistant to formally verify token smart contracts and provide a
testing tool for developers of such contracts.</p>
</li>
<li>
<p><a href="https://www.nomadic-labs.com/careers/generation-of-scenario-tests/">Generation of Scenario
Tests</a>:
build a syntax extension tool that generates tests scenarios and reports from
OCaml functions specifications.</p>
</li>
<li>
<p><a href="https://www.nomadic-labs.com/careers/live-monitoring-of-tezos-nodes-tezos-metrics/">Tezos-metrics, Live Monitoring of Tezos
Nodes</a>:
enhance an existing monitoring tool and add live analysis of the protocol execution.</p>
</li>
<li>
<p><a href="https://www.nomadic-labs.com/careers/mechatez-formally-verifying-critical-features-of-tezos-protocols/">MechaTez, Formally Verifying Critical Features of Tezos
Protocols</a>:
extend existing Coq formal verification of the protocol to target new features</p>
</li>
<li>
<p><a href="https://www.nomadic-labs.com/careers/integrating-static-analysis-in-smart-contract-development-tools/">Integrating static analysis in smart contract development
tools</a>:
build an lsp based interface for michelson and brought it to actual development tool</p>
</li>
<li>
<p><a href="https://www.nomadic-labs.com/careers/how-to-reason-on-traces-between-tezos-nodes/">How to Reason on Traces between Tezos
Nodes</a>:
build a logical framework to analyze traces of messages exchanged by tezos nodes</p>
</li>
<li>
<p><a href="https://www.nomadic-labs.com/careers/contribute-to-the-next-tezos-blockchain-protocol-amendment/">Contribute to the next Tezos blockchain protocol
amendment</a>:
build a new feature for the protocol, from start to finish.</p>
</li>
<li>
<p><a href="https://www.nomadic-labs.com/careers/react-reason-programming-on-a-wallet-application-in-a-blockchain-setting/">React/Reason programming on a wallet application in a blockchain
setting</a>:
take part on the development of the umami-wallet using React and ReasonMl.</p>
</li>
<li>
<p><a href="https://www.nomadic-labs.com/careers/improve-our-formal-verification-framework/">Improve our Formal Verification
Framework</a>:
design and implement formal tools in an industrial setting.</p>
</li>
<li>
<p><a href="https://www.nomadic-labs.com/careers/comparing-program-proof-tools-for-ocaml-programs-in-an-industrial-setting/">Comparing Program Proof Tools for OCaml Programs in an Industrial
Setting</a>: benchmark formal verification tools for OCaml, and experiment especially with FreeSpec.</p>
</li>
</ul>
<p><a id="own"></a></p>
<p>You can also propose your own project, provided it is close to Tezos or the work we do at Nomadic Labs. Please just describe your interests and outline your ideal internship, and we will be happy to discuss developing the topic together.</p>
<p>To submit your application, please send an email to <a href="mailto:internship@nomadic-labs.com">internship@nomadic-labs.com</a>,
or click or follow the link of the desired subject and fill in the form at the bottom of the page.</p>
<p><em>Are you a lecturer who would like to share these internships with your students?</em>
Thank you sincerely for your interest: please feel free to share our <a href="https://www.nomadic-labs.com/wp-content/uploads/2021/09/2021-2022_internship_catalog.pdf">2021-22 internship catalog</a>.</p>Three questions to Nomadic Labs apprentices — Killian Delarue2021-08-27T14:00:00+02:002021-08-27T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-08-27:/three-questions-to-nomadic-labs-apprentices-killian-delarue.html<p>Short interview with Killian Delarue</p><p>Nomadic Labs is delighted to host apprentices (<em>apprentis</em>).
These are students — usually studying for an undergraduate or Master’s degree — who work with us in parallel with their studies for a period of one to two years.
Each apprentice has a unique story to tell about how their understanding of blockchain as a practical and applied technology has developed during their time with us.</p>
<p>In this blogpost, we will ask three questions of one of our current apprentices: <strong>Killian Delarue</strong> (and a couple of questions of his mentor at Nomadic Labs).</p>
<p>Killian — so happy to have you with us!
We hope you’re having a wonderful and educational period with Nomadic Labs.
We’d love to hear a bit about you and your activities <em>chez nous</em> …</p>
<h1 id="questions-for-killian"><a class="toclink" href="#questions-for-killian">Questions for Killian</a></h1>
<h2 id="1-please-present-yourself-and-your-academic-background"><a class="toclink" href="#1-please-present-yourself-and-your-academic-background">1. Please present yourself and your academic background</a></h2>
<p>My name is Killian Delarue, I’m a 24 year old student and an apprentice at Nomadic Labs.</p>
<p>I’m in the final year of a five-year degree in computer science and software engineering at <a href="https://www.ensiie.fr/"><span class="caps">ENSIIE</span></a> graduating in September 2021. My subject specialization is functional programming and formal methods.</p>
<p>In summer 2020 I did a three-month internship at Nomadic Labs, and I was very pleased in September 2020 to return on a professional development contract. This means that I get to work on a project with Nomadic Labs, in parallel with the final year of my engineering studies. The contract will continue until my graduation in September — so it’s a full year of apprenticeship with one of the best companies in the sector!</p>
<h2 id="2-tell-us-more-about-your-apprenticeship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months"><a class="toclink" href="#2-tell-us-more-about-your-apprenticeship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months">2. Tell us more about your apprenticeship: main subject, who is your mentor, what you have learned and especially, what surprised you the most within these months?</a></h2>
<p>I’m working with the Node and Tooling team, mentored by Julien Tesson on a project to build a user interface to launch and interact with a Tezos node on the terminal.
My project aims to improve the user experience of Tezos.
It provides a clean, easy and flexible way to run and monitor a Tezos node on your terminal.</p>
<p>My project requires me to understand the architecture of Nomadic Labs’ software and to carry out an efficient technical development within it.
I already had some familiarity with the basics of functional programming (especially OCaml) from my engineering school, but this project has taught me more advanced aspects of programming, and more about the OCaml toolstack. </p>
<p>This project has also helped me to develop new communication skills, as it requires close coordination with the development teams of Nomadic Labs.
We have to discover, understand, and take into account the needs and suggestions of the developers and users, and conversely I have to communicate back to them about the progress of the project.</p>
<p>Working at Nomadic Labs has been an opportunity to be part of a team of awesome people with great human values and a great attitude to mentorship.
Undergraduate courses are done by individuals but software is created by teams, so it has been a delight to learn how important it is to be technically competent <em>and also</em> well-integrated working with members of the team.</p>
<h2 id="3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-apprenticeship"><a class="toclink" href="#3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-apprenticeship">3. Why did you choose to become a part of the blockchain ecosystem and Nomadic labs? What are your plans after completing this apprenticeship?</a></h2>
<p>Before my 2020 summer internship at Nomadic Labs and my year’s apprenticeship since then, I didn’t know much about blockchain.
I heard about Nomadic Labs and Tezos from a teacher at school.
I started to read Tezos project documentation and found myself fascinated by the algorithms, the techniques, the libraries used, and in general the detailed software architectures that help achieve robustness in a large-scale safety-critical distributed system.
Also, I like that it is open-source and uses OCaml.</p>
<p>So I was delighted to get an apprenticeship with Nomadic Labs to develop my technical skills while learning more about OCaml, the blockchain ecosystem, and about Tezos in particular.</p>
<p>I have found the experience thrilling and hope to keep working in the OCaml community, and hopefully within the blockchain ecosystem.</p>
<h1 id="questions-for-killians-nomadic-labs-mentor-julien-tesson"><a class="toclink" href="#questions-for-killians-nomadic-labs-mentor-julien-tesson">Questions for Killian’s Nomadic Labs mentor Julien Tesson.</a></h1>
<h2 id="what-is-your-input-to-the-work-carried-out-by-killian"><a class="toclink" href="#what-is-your-input-to-the-work-carried-out-by-killian">What is your input to the work carried out by Killian?</a></h2>
<p>We started the project from scratch at the beginning of Killian’s apprenticeship back in September 2020, and he has become highly committed to it.
Killian did a very good job of designing his prototype, and we have had some great discussions on the project architecture and on the user experience offered.</p>
<p>During the year of Killian’s apprenticeship it’s been my pleasure to watch Killian learn and evolve as a software engineer, and I have been impressed by his determination and his progress.
His apprenticeship with us will end soon, when he graduates, and I wish Killian all the best for his future!</p>
<h2 id="why-is-the-topic-of-this-apprenticeship-important"><a class="toclink" href="#why-is-the-topic-of-this-apprenticeship-important">Why is the topic of this apprenticeship important?</a></h2>
<p>Currently, you can download <a href="https://gitlab.com/tezos/tezos/">Octez</a> the Nomadic Labs implementation of Tezos, and fire up the command <code>tezos-node</code> to run a Tezos node (<a href="https://tezos.gitlab.io/introduction/howtorun.html">tutorial here</a>). And this is all good — but even a professional developer would agree that the <code>tezos-node</code> logging output is not the easiest to read.</p>
<p>A command-line client exists which can query a node for more information, but it does not necessarily deliver more easy-to-read information. Developers also have specific tools to monitor node health — you launch a dedicated program to query the node, that feeds a <a href="https://prometheus.io">Prometheus service</a> that itself feeds a <a href="https://grafana.com/">Grafana service</a>, which serves you a webpage.
This is powerful, but not necessarily well-suited to every end-user.</p>
<p>Killian is developing a <em>lightweight</em> and <em>user-friendly</em> standalone prototype control utility with which a user can run a node and interact with it, getting meaningful and easily-digested diagnostic information from the node, and being able to issue commands via the utility to control the node’s behaviour. Think: a <em>“a dashboard, for Tezos nodes”</em>.</p>
<p>This is quite a large project that would greatly improve the user experience, and Killian’s prototype has laid the foundations for it.
I am very pleased with his work.</p>Three questions to Nomadic Labs apprentices — Daniel Jean2021-08-20T14:00:00+02:002021-08-20T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-08-20:/three-questions-to-nomadic-labs-apprentices-daniel-jean.html<p>Short interview with Daniel Jean</p><p>Nomadic Labs is delighted to host apprentices (<em>apprentis</em>).
These are students — usually studying for an undergraduate or Master’s degree — who work with us in parallel with their studies for a period of one to two years.
Each apprentice has a unique story to tell about how their understanding of blockchain as a practical and applied technology has developed during their time with us.</p>
<p>In this blogpost, we will ask three questions of one of our current apprentices: <strong>Daniel Jean</strong> (and a couple of questions of his mentor at Nomadic Labs).</p>
<p>Daniel — so happy to have you with us!
We hope you’re having a wonderful and educational period with Nomadic Labs.
We’d love to hear a bit about you and your activities <em>chez nous</em> …</p>
<h1 id="questions-for-daniel"><a class="toclink" href="#questions-for-daniel">Questions for Daniel</a></h1>
<h2 id="1-please-present-yourself-and-your-academic-background"><a class="toclink" href="#1-please-present-yourself-and-your-academic-background">1. Please present yourself and your academic background</a></h2>
<p>I am Daniel Jean.
I’m in the first year of a two-year Master’s degree in financial engineering at <a href="https://www.esilv.fr/"><span class="caps">ESILV</span></a>, with a specialisation in fintech.
From 2018-2020 I did a three-year Bachelor’s degree in computing at the <a href="https://u-paris.fr/en/">Université de Paris</a>.</p>
<p>I am also an enthusiastic Judo player.
I started when I was three and represented France at the 2018 European and World championships in Israel and Azerbaijan.</p>
<h2 id="2-tell-us-more-about-your-apprenticeship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months"><a class="toclink" href="#2-tell-us-more-about-your-apprenticeship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months">2. Tell us more about your apprenticeship: main subject, who is your mentor, what you have learned and especially, what surprised you the most within these months?</a></h2>
<p>With the growing adoption of blockchain, we are witnessing vigorous interest from users, companies, banks, and institutions for the technology in general and for the (rapidly developing) Tezos blockchain in particular. </p>
<p>Tezos has some unique features which in principle make it well-suited to industrial adoption: it’s fully open-source; it has an on-chain governance structure; and it’s particularly flexible with its regular auto-update feature.
Indeed, Tezos builds in upgrade propositions and an associated voting mechanism which allows users to vote for or against proposed updates in a fully transparent manner.</p>
<p>This is fine in high-level terms — but blockchain is an industry that is very new and very competitive.
So concretely, we still need to communicate the benefits and potential of the Tezos ecosystem, and discuss with companies like Exaion, Ubisoft and Société Générale how integration with Tezos software can meet their current and future business needs.
Conversely, we also need to understand our users’ needs and feed these back to Nomadic Labs’ developers. </p>
<p>This process of communication and consensus-building is vital, and that’s why a good support team is needed.
It’s never just about the tech; it’s always also about how that tech is communicated, developed, and supported.</p>
<p>As Nomadic Labs’ support team, we help users to experiment, test, and build on the Tezos blockchain.
We design case studies and proofs of concept, and support the creation and deployment of solutions to the Tezos main network.
We help users to solve issues, and create documentation to explain the core principles of Tezos and so make it easy for newcomers to join the Tezos ecosystem.</p>
<p>My mentor is Sébastien Choukroun.
He has been a very helpful and engaged supervisor and I have learned a lot from him and from the rest of the support team.</p>
<p>I have been working at Nomadic Labs since August 2020.
It’s been very fruitful to strengthen my knowledge of blockchain in general and deepen my expertise in Tezos in particular.
I have learned about the core features of Tezos and how the different components of this complex software work.
My discussions with institutional users have taught me how to effectively listen to users’ needs and deliver effective support; be this about helping to launch Tezos nodes, new bakers, or reviewing smart contracts. </p>
<p>Tezos brings the third pillar of blockchain technology to the industry: the auto-evolutivity through its governance core principle called <a href="https://www.tezosagora.org/learn">self-amendment</a>.
Tezos is capable of evolving every three months to improve its software and make it more robust, scalable and user-friendly.
Self-amendment, and the technical robustness which it implies, can be a key differentiator to create a sustainable network and secure long-term adoption. </p>
<p>For instance, I was delighted when Tezos users recently <a href="https://tzstats.com/election/29">accepted</a> the <a href="https://tezos.gitlab.io/protocols/010_granada.html">Granada amendment of the protocol</a>, bringing various improvements to ensure a better scalability, liquidity of the <span class="caps">XTZ</span> (the native coin of Tezos), and gas optimisations (related to fees paid when performing operations on the blockchain).</p>
<h2 id="3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-apprenticeship"><a class="toclink" href="#3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-apprenticeship">3. Why did you choose to become a part of the blockchain ecosystem and Nomadic labs? What are your plans after completing this apprenticeship?</a></h2>
<p>I heard about blockchains three years ago after the 2018 crash, and I was curious about this new geek “thing” to allow people to exchange value and expand financial technology through smart contracts.
I started to read about blockchain technology and before I knew it, I had fallen into the rabbit hole (as we say in the ecosystem).
I’ve been happily in it ever since.</p>
<p>I learned about the basic principles of Tezos and found the blockchain really interesting, technically speaking.
During a discussion with a friend, I discovered that Nomadic Labs was working on Tezos and that It could be a great place to learn and deepen my knowledge of the software.
I jumped at the chance to apply at Nomadic Labs … and here I am!</p>
<p>After my apprenticeship, I plan to further invest myself in the blockchain industry, hopefully as an engineer at Nomadic Labs.
Also, I am delighted to run a Tezos baker <a href="https://www.baking.finance/">Baking.Finance</a> and I plan to keep maintaining it in the future. </p>
<h1 id="questions-for-daniels-nomadic-labs-mentor-sebastien-choukroun"><a class="toclink" href="#questions-for-daniels-nomadic-labs-mentor-sebastien-choukroun">Questions for Daniel’s Nomadic Labs mentor Sébastien Choukroun</a></h1>
<h2 id="what-is-your-input-to-the-work-carried-out-by-daniel"><a class="toclink" href="#what-is-your-input-to-the-work-carried-out-by-daniel">What is your input to the work carried out by Daniel?</a></h2>
<p>I helped Daniel to focus on priorities amongst the projects he worked on.
I also gave him feedback on project management.
This provided direction for his answers to users and for preparing materials for Tezos users.
I also shared advice on good practice for blockchain projects with financial institutions.</p>
<h2 id="why-is-the-topic-of-this-apprenticeship-important"><a class="toclink" href="#why-is-the-topic-of-this-apprenticeship-important">Why is the topic of this apprenticeship important?</a></h2>
<p>It helped users to interact with Tezos, in particular Exaion, Ubisoft and Société Générale.
His experience as a baker allowed him to share good practice and handle potential issues faced by these corporate bakers.</p>Three questions to Nomadic Labs interns — Julien Coolen2021-08-16T14:00:00+02:002021-08-16T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-08-16:/three-questions-to-nomadic-labs-interns-julien-coolen.html<p>Short interview with Julien Coolen</p><p>Each year we host around half-a-dozen interns (<em>stagiaires</em>), for periods of two to six months.
These young men and women are usually studying or have just finished studying for undergraduate or Master’s degrees, and are eager to gain experience in the blockchain industry.
Each has a unique story to tell about their individual interest in this technology.</p>
<p>In this blogpost, we will ask three questions of one of our current interns: <strong>Julien Coolen</strong> (and a couple of questions of his mentors at Nomadic Labs).</p>
<p>Julien — so happy to have you with us!
We hope you’re having a wonderful and educational period with Nomadic Labs.
We’d love to hear a bit about you and your activities <em>chez nous</em> …</p>
<h1 id="questions-for-julien"><a class="toclink" href="#questions-for-julien">Questions for Julien</a></h1>
<h2 id="1-please-present-yourself-and-your-academic-background"><a class="toclink" href="#1-please-present-yourself-and-your-academic-background">1. Please present yourself and your academic background</a></h2>
<p>I am a first-year master’s student in cryptology (<a href="https://web.archive.org/web/20210512050031/https://master.math.univ-paris-diderot.fr/annee/m1-mi/">double master Mathématiques, Informatique de la cryptologie et sécurité</a>) at the Université de Paris. I graduated in 2020 with bachelor’s degrees in Mathematics and Computer Science.</p>
<p>Being fond of <a href="https://en.wikipedia.org/wiki/Open_research">open research</a>, <a href="https://en.wikipedia.org/wiki/Open_source">open source</a>, and <a href="https://en.wikipedia.org/wiki/Formal_verification">formal verification</a>, I tried to gain experience in these areas.
In 2019 I worked as a system administrator intern to configure and deploy a digital library for my mathematics faculty.
Then in 2020 a friend and I attempted to program and prove the correctness of an <span class="caps">LL</span>(1) parser using <a href="https://coq.inria.fr/">the Coq proof assistant</a>.
We thank Yann Régis-Gianas for this learning opportunity.
Also in 2020, I developed features for <a href="https://github.com/coq/bot">an OCaml bot</a> to simplify collaborative software development, under the supervision of Théo Zimmermann at <a href="https://www.inria.fr/en">Inria</a>.</p>
<h2 id="2-tell-us-more-about-your-internship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months"><a class="toclink" href="#2-tell-us-more-about-your-internship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months">2. Tell us more about your internship: main subject, who is your mentor, what you have learned and especially, what surprised you the most within these months?</a></h2>
<p>I am developing a distributed hash table (<span class="caps">DHT</span>) for file sharing during a three-month internship from June to August 2021, under the supervision and mentorship of research engineers Vivien Pelletier and Julien Tesson at Nomadic Labs.</p>
<p>A hash table is a key-value store; you need a key to retrieve each piece of stored information.
However as your store grows you become limited by computer memory.
The <em>distributed</em> nature of DHTs addresses this issue: the data is distributed around different computers on the internet.
An example of a <span class="caps">DHT</span> is the <a href="https://ipfs.io/"><span class="caps">IPFS</span> distributed web</a>. </p>
<p>There are several ways to implement a <span class="caps">DHT</span>.
We have chosen one that exploit the Tezos peer-to-peer (<span class="caps">P2P</span>) library as an off-the-shelf and well-tested component.
With it, the distributed parts of the distributed hash table (called <strong>peers</strong>) can communicate.</p>
<p>This project aims to test the architecture of the peer-to-peer library of Tezos in these somewhat different circumstances:</p>
<ol>
<li>In the Tezos blockchain, connections between peers are arbitrary, whereas in a <span class="caps">DHT</span> connections follow specific patterns and rules. </li>
<li>
<p>The size of a message sent across the Tezos network is typically a few kilobytes, and only occasionally megabytes.
The Tezos network protocol never requires to transfer a message more than 100 megabytes; meanwhile the serialization library (which interfaces between the Tezos network protocol and the peer-to-peer library itself) is limited to a message size of 1 gigabyte.</p>
<p>My <span class="caps">DHT</span> places no theoretical limit on message size (though there are practical ones, e.g. disk space) so we intend to benchmark and then optimize performance of the peer-to-peer network using my <span class="caps">DHT</span>, to see how far we can scale practical message size.
We hope to attain at least gigabytes — thus ensuring that the practical capabilities of the peer-to-peer library are a thousandfold above that required by the exigencies of the Tezos network.</p>
</li>
</ol>
<p>After a bit more than a month, I (with help from my mentors) have built a prototype of <span class="caps">DHT</span> and tested it using <a href="https://tezos.gitlab.io/developer/tezt.html">the library for unit and integration testing</a> from Tezos.
To achieve this, I had to familiarize myself with part of <a href="https://gitlab.com/tezos/tezos/">the Octez codebase</a>, which is the Tezos implementation to which Nomadic Labs contributes.</p>
<p>I thank Vivien and Julien for explaining the innards of a full-scale industrial codebase running live and safety-critical code.
This experience has also brought home to me how just one tiny mistake in a distributed system can corrupt the entire network!</p>
<h2 id="3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-internship"><a class="toclink" href="#3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-internship">3. Why did you choose to become a part of the blockchain ecosystem and Nomadic labs? What are your plans after completing this internship?</a></h2>
<p>I chose Nomadic Labs because the company applies formal methods to build robust, open-source software with the <a href="https://ocaml.org/">OCaml</a> programming language.
In particular, their smart-contract stack-based language <a href="https://www.michelson.org/">Michelson</a> is the only one of its kind to be <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/">formalized</a>!</p>
<p>Blockchains are in an exploratory phase, so they raise many technical challenges and create many opportunities for innovation and discovery.
The end goal is to build resilient, transparent, and accessible systems.</p>
<p>After this internship, I hope to continue working with functional programming languages and formal methods.</p>
<h1 id="questions-for-juliens-nomadic-labs-mentors-vivien-pelletier-and-julien-tesson"><a class="toclink" href="#questions-for-juliens-nomadic-labs-mentors-vivien-pelletier-and-julien-tesson">Questions for Julien’s Nomadic Labs mentors Vivien Pelletier and Julien Tesson.</a></h1>
<h2 id="what-is-your-input-to-the-work-of-julien"><a class="toclink" href="#what-is-your-input-to-the-work-of-julien">What is your input to the work of Julien?</a></h2>
<p>Julien is independent, and in a good way.
Our supervision consists mostly of discussing objectives for the week and answering his many penetrating technical questions.</p>
<h2 id="why-is-the-topic-of-this-internship-important"><a class="toclink" href="#why-is-the-topic-of-this-internship-important">Why is the topic of this internship important?</a></h2>
<p>This peer-to-peer layer is tailored to the OCaml Tezos node implementation (which is now called <a href="https://gitlab.com/tezos/tezos/">Octez</a>).
By using this layer for other purposes — sharing files, in this case — we hope to improve its maturity.</p>
<p>In developing a distributed hash table, Julien had to add some features to this layer that should be useful for future improvements of the Tezos node; for example, the possibility to query the related network address of a peer, given its cryptographic identity.
Another feature that Julien plans to add is to let users of this layer configure the topology to their needs.</p>Network Updates from the Granada Protocol Amendment2021-08-13T08:01:00+02:002021-08-13T08:01:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-08-13:/updates-granada.html<p>Network Updates from the Granada Protocol Amendment</p><p>The Granada protocol amendment was activated at block 1589248 on Friday 6 August 2021 at <a href="https://tzstats.com/1589248">11:36 <span class="caps">AM</span> <span class="caps">CEST</span></a>. Nomadic Labs would like to thank the community, bakers, and development partners for their diligent assistence in developing and adopting this proposal. </p>
<p>On May 31st, Nomadic Labs proposed together with Marigold, Oxhead Alpha, Tarides, and DaiLambda <a href="https://research-development.nomadic-labs.com/announcing-granada.html">an amendment to the Tezos protocol named Granada</a>. Tezos’ on-chain governance procedure enabled the adoption of the proposal by the community and the subssequent activation of Granada.</p>
<p>Granada introduced three important upgrades:</p>
<ul>
<li>Liquidity Baking: 2.5 tez are minted at each block to incentivize liquidity providers of a tzBTC/tez pair</li>
<li>Emmy*: a change in the consensus algorithm making blocks up to twice more frequent</li>
<li>Gas improvements coming from a refactoring of the Michelson interpreter and an optimization of data serialization.</li>
</ul>
<p>Since August 10, 2021 23:19 <span class="caps">CEST</span>, the Granada protocol has run on Tezos mainnet for a complete cycle (8192 blocks). Below, Nomadic Labs is pleased to share data from the first complete cycle and an analysis of the impact of these three features.</p>
<h2 id="liquidity-baking"><a class="toclink" href="#liquidity-baking">Liquidity Baking</a></h2>
<p>Liquidity baking is a primary example of the utility of Tezos’ on-chain governance to provide for public goods that go beyond securing the network.</p>
<p>The liquidity baking proposal is an experiment to see if a decentralised protocol can use incentives to create liquidity around them. As of publication, over 980,000 tez and 72.3 tzBTC (about $6.8M worth) were deposited in the liquidity baking contract, demonstrating a successful deployment of the contract and concept. </p>
<p>During the first cycle of Granada, total value locked in the liquidity baking contract steadily increased. In the first few hours after going live, over 2 million <span class="caps">USD</span> worth of coins (24.19 tzBTC + 331,000 tez) were deposited. </p>
<p><img alt="Graph of the so-called "TVL", courtesy of liquidity-baking.com" src="https://i.imgur.com/dMHkUdY.png">
<em>Graph of the so-called “<span class="caps">TVL</span>”, courtesy of liquidity-baking.com</em></p>
<p>Community members can track live statistics on liquidity baking on the statistics page developed by Tessellated Geometry — <a href="http://liquidity-baking.com/mainnet.html">liquidity-baking.com</a>.</p>
<h2 id="emmy"><a class="toclink" href="#emmy">Emmy*</a></h2>
<p>Emmy<em> consensus algorithm is one of the most important changes to the core protocol of Tezos in Granada. Emmy</em> tweaked the block delay formula to allow blocks to be produced more frequently. </p>
<p>Emmy* is the culmination of multiple updates to the consensus algorithm. In <a href="https://research-development.nomadic-labs.com/emmy-seven-years-of-updatable-consensus.html">a recent survey</a> we have detailed the various flavours of the Emmy family of consensus algorithms.</p>
<p>Before Granada, the minimum delay between blocks on Tezos mainnet was set to 60 seconds. When the network was perfectly healthy (with all bakers and endorsers online and able to quickly produce, validate, and transmit blocks and endorsements), the chain advanced at the speed of one block every minute. When network health dropped however, the chain reacted to maintain consensus. This works based on the number of endorsements that a baker includes in a given block; each block contained 32 endorsement slots and if less than 2/3 of these slots were filled, the baker had to wait before publishing its block (the fewer slots are filled, the longer the delay).</p>
<p>Emmy* does not fundamentally change this behavior, but it changes the parameters in order to decrease the latency of the chain. In Granada, each block contains 256 endorsing slots (instead of 32), the minimal delay between blocks is 30 seconds (instead of 60), and the required proportion of endorsement slots needed to publish a block after 30 seconds is 60%.</p>
<p><a name="image"></a></p>
<p><img alt="Figure 1" src="https://research-development.nomadic-labs.com/images/levels2.png"></p>
<p>Figure 1 shows the level of each block of the last cycle of Florence (cycle 387) and the first cycle of Granada (cycle 388) in function of the time the block was produced.</p>
<h3 id="block-times"><a class="toclink" href="#block-times">Block Times</a></h3>
<p>Activation of Granada and Emmy* caused a temporary slow down in the chain. Blocks were produced on average every 2 minutes and 11 seconds with a maximum time between blocks of roughly 15 minutes.</p>
<p>Our investigations uncovered several issues. First, blocks that arrived too late were not endorsed at all, causing the next block to be produced after the maximal block delay (almost 15 minutes for a block baked at priority 0). <a href="https://forum.tezosagora.org/t/releasing-version-9-6-of-octez/3543">Version 9.6 of Octez</a> was released on Friday at about <span class="caps">5PM</span> <span class="caps">CEST</span> to mitigate the delay. The update increased the delay after which the endorser gives up on endorsing to 1200 seconds (previously 110 seconds).</p>
<p>Version 9.6 of Octez ensured no blocks were baked completely without endorsements, solving a key aspect of the slow down. After further research, we noticed numerous computations took place during the handling of pending consensus operations. Fixes for this problem were included in <a href="https://forum.tezosagora.org/t/releasing-version-9-7-of-octez/3547">version 9.7 of Octez</a>, released on Saturday August 7 at about 10 <span class="caps">PM</span> <span class="caps">CEST</span>. </p>
<p>Bakers reacted quickly and updated to the new versions of Octez. This time, the rate at which blocks are produced and propagated on Mainnet was significantly improved: quickly after the version 9.7 of Octez release the chain average delay between blocks decreased to almost 30 seconds.</p>
<p>If we take as reference the block production frequency of mainnet until the activation of Granada (one block per minute), it took 3 days, 6 hours, and 51 minutes for the chain to catch up at <a href="https://tzstats.com/1593978">block 1593978</a> (the 4731 first blocks of Granada were produced in 4731 minutes).</p>
<h3 id="remaining-issues"><a class="toclink" href="#remaining-issues">Remaining Issues</a></h3>
<p>For the rest of the cycle, the chain has run at the expected speed but problems remain: endorsements are still being missed more often than expected. We are continuing to monitor the situation and to investigate the potential causes of the missing endorsements. </p>
<p>At the moment, we are leaning towards the conclusion that with Emmy* block propagation itself may be reaching the limits of what some bakers are capable to sustain. We are thus investigating several ways to improve block propagation, with possible optimizations such as only validating block headers before propagating blocks. </p>
<p>We would like to thank the community for their continuous support. We are especially grateful to everyone that promptly upgraded their nodes, tested patches and provided feedback. The collaboration and communication of the community and bakers contributed to our ability to identify and mitigate the network issues. </p>
<p>If you are running a node and haven’t upgraded it to version 9.7 yet, we strongly advise you to do so.</p>
<h2 id="gas-improvements"><a class="toclink" href="#gas-improvements">Gas improvements</a></h2>
<p>As we explained <a href="https://research-development.nomadic-labs.com/a-tale-of-two-reductions-in-gas-consumption-in-tezos.html">in a recent post</a>, Granada includes two important gas optimizations: a refactoring of the Michelson interpreter and memoization of the serialization function for recursive types. </p>
<p>In order to evaluate the impact of these gas improvements we have measured the gas consumption of all contract calls in the last cycle of Florence (cycle 387) and in the first cycle of Granada (cycle 388). There is lot of variation between the gas consumed in the smart contract calls because the called contracts can be very different. To make the result smoother and more readable, we have averaged these gas consumption measured over a span of 1000 smart contract calls. </p>
<p>The result can be seen in the figure below:</p>
<p><a name="image"></a></p>
<p><img alt="Figure 1" src="https://research-development.nomadic-labs.com/images/gas2.png"></p>
<p>Overall, average gas per smart contract call has been decreased by a factor of 5; from 43197 gas units per call in cycle 387 to 8591 gas units per call in cycle 388.</p>
<h2 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h2>
<p>Thanks to the activation of the Granada amendment, more than 5 million <span class="caps">USD</span> worth of liquidity have been provided to the newly deployed Liquidity Baking contract, the Tezos chain is moving almost twice as fast as before, and the gas consumption for smart contract calls has been reduced by a factor of 5. </p>
<p>We acknowledge the migration was not as smooth as usual. All Tezos dev teams understand the inconvenience that results from slowing down of the chain and missed endorsements. This was caused by a combination of issues, some of which still require investigation. We would like to thank the responsiveness of community members and bakers who applied emergency Octez updates. </p>Three questions to Nomadic Labs interns — Valentin Chaboche2021-08-06T14:00:00+02:002021-08-06T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-08-06:/three-questions-to-nomadic-labs-interns-valentin-chaboche.html<p>Short interview with Valentin Chaboche</p><p>Each year we host around half-a-dozen interns (<em>stagiaires</em>), for periods of two to six months.
These young men and women are usually studying or have just finished studying for undergraduate or Master’s degrees, and are eager to gain experience in the blockchain industry.
Each has a unique story to tell about their individual interest in this technology.</p>
<p>In this blogpost, we will ask three questions of one of our current interns: <strong>Valentin Chaboche</strong> (and a couple of questions of his mentors at Nomadic Labs).</p>
<p>Valentin — so happy to have you with us!
We hope you’re having a wonderful and educational period with Nomadic Labs.
We’d love to hear a bit about you and your activities <em>chez nous</em> …</p>
<h1 id="questions-for-valentin"><a class="toclink" href="#questions-for-valentin">Questions for Valentin</a></h1>
<h2 id="1-please-present-yourself-and-your-academic-background"><a class="toclink" href="#1-please-present-yourself-and-your-academic-background">1. Please present yourself and your academic background</a></h2>
<p>My name is Valentin Chaboche, I am 23 years old and a student at the Université de Paris. </p>
<p>In 2019 I graduated from the Université Paris Diderot 7 with a bachelor’s degree in Computer Science, then I stayed for a <a href="http://www.informatique.univ-paris-diderot.fr/formations/masters/langages_et_programmation/accueil">master’s degree in <em>Language et Programmation</em></a>, graduating in 2021.
The course content includes algorithms and programming, distributed systems, and formal verification. </p>
<h2 id="2-tell-us-more-about-your-internship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months"><a class="toclink" href="#2-tell-us-more-about-your-internship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months">2. Tell us more about your internship: main subject, who is your mentor, what you have learned and especially, what surprised you the most within these months?</a></h2>
<p>I am doing an internship at Nomadic Labs with the verification team under the supervision of Arvid Jakobsson and Zaynah Dargaye, from March to September 2021.</p>
<p>Nomadic labs is always working to improve the quality of the Tezos codebase and assure its correct behaviour — in particular through mass testing of the codebase and reduction of boilerplate code.
My internship aims to contribute as follows:</p>
<ol>
<li>Develop an annotation-based system for specifying invariants of OCaml functions. </li>
<li>Implement a preprocessor that transforms the annotations to property-based tests.</li>
<li>Explore heuristics for automatically constructing data-generators for types of particular relevance to the Octez codebase (the OCaml Tezos implementation).</li>
</ol>
<p>My internship has exposed me to many new ideas, including formal verification of programs, blockchains and distributed systems, and metaprogramming.
This is also my first experience in a collaborative open-source project.
During my internship I have had the opportunity to develop and release libraries, create and review merge requests in Tezos, and participate in open-source projects used in Tezos.
I had to learn to communicate and transfer knowledge about my work at Nomadic Labs within development teams, through meetings in small groups, documents, and team presentations.</p>
<p>During the three months of my internship so far, I have been surprised and delighted by the kindness and goodwill of people at Nomadic Labs and in the OCaml ecosystem.
They have always taken the time to help me with technical and theoretical questions.
I’ve found it to be a welcoming environment for a newcomer like me and I am grateful to have had an opportunity to be part of such a great community.</p>
<h2 id="3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-internship"><a class="toclink" href="#3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-internship">3. Why did you choose to become a part of the blockchain ecosystem and Nomadic labs? What are your plans after completing this internship?</a></h2>
<p>I strongly believe in distributed and decentralized systems, so participating in such a complex and rich project was a great opportunity for me!
Nomadic Labs hosts very smart engineers — and I have discovered that a lot of theoretical insight goes into designing Tezos. </p>
<p>This internship has been an opportunity to work with and learn from talented software engineers and researchers.
It’s the perfect opportunity for an OCaml enthusiast such as myself to participate in the development of a complex OCaml codebase, with sophisticated programming features, all while working on one of the leading projects of the OCaml ecosystem.</p>
<h1 id="questions-for-valentins-nomadic-labs-mentors-arvid-jakobsson-and-zaynah-dargaye"><a class="toclink" href="#questions-for-valentins-nomadic-labs-mentors-arvid-jakobsson-and-zaynah-dargaye">Questions for Valentin’s Nomadic Labs mentors Arvid Jakobsson and Zaynah Dargaye</a></h1>
<h2 id="what-is-your-input-to-the-work-of-valentin"><a class="toclink" href="#what-is-your-input-to-the-work-of-valentin">What is your input to the work of Valentin?</a></h2>
<p>Valentin already had good familiarity with algorithms and programming, and with the development and versioning tools which we use at Nomadic Labs.
Our role was to teach him how industrial research and development is carried out: study the available state of the art, consider and select a solution that seems likely to work, and develop it incrementally as directed by relevant use cases.</p>
<h2 id="why-is-the-topic-of-this-internship-important"><a class="toclink" href="#why-is-the-topic-of-this-internship-important">Why is the topic of this internship important?</a></h2>
<p>At Nomadic Labs, we are constantly looking to assure the high quality of our code.</p>
<p><strong>Property-based testing</strong> is a lightweight formal methods-based style of testing that lies between formal methods and unit tests.
It relies on asserting correctness properties of code, and then using the computer’s own computational power to generate huge numbers of unit tests from test script heuristics which, in practice, are highly effective at finding corner cases — often more effective than humans. </p>
<p>Our experience with property-based testing has revealed some practical bottlenecks to its more widespread application.
In particular, we have found that writing input-set generators and test-scripts can be time-consuming, requires expertise in the tooling, and can become a source of bugs. </p>
<p>Valentin’s internship is exploring techniques to automate such steps, and so permit a more widespread practice of property-based testing in the validation phase of developments in Nomadic labs, and more generally for developments in the OCaml language.
This would assure a higher-quality codebase while still making relatively light demands on our developers to provide information for generating property-based tests. </p>
<p>Valentin’s first results are promising.
He has been able to validate his approach on concrete use cases in our team’s developments, and we expect other software components could soon benefit from it too.</p>Granada, the latest Tezos upgrade, is LIVE2021-08-06T08:00:00+02:002021-08-06T08:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-08-06:/granada-the-latest-tezos-upgrade-is-live.html<p><em>This is a joint announcement from Nomadic Labs, Marigold, Oxhead Alpha, Tarides and DaiLambda.</em></p>
<p>On 6 August 2021, the Tezos blockchain successfully upgraded by adopting Granada at block #1,589,248. Granada was jointly developed by <a href="https://www.nomadic-labs.com/">Nomadic Labs</a>, <a href="https://marigold.dev/">Marigold</a>, <a href="https://www.oxheadalpha.com/">Oxhead Alpha</a>, <a href="https://tarides.com/">Tarides</a> and <a href="https://www.dailambda.jp/">DaiLambda</a>. It follows the <a href="https://tzstats.com/protocols/PsFLorenaUUuikDWvMDr6fGBRG8kt3e3D3fHoXK1j1BFRxeSH4i">Florence protocol upgrade</a> three months ago, and will be the seventh Tezos upgrade since Athens’ activation in May 2019 (overviews <a href="https://tzstats.com/protocols">here</a> and <a href="https://tezos.gitlab.io/protocols/naming.html">here</a>).</p>
<p>Granada includes <a href="https://tezos.gitlab.io/protocols/010_granada.html">several bug fixes and small improvements</a> and includes the following substantive changes:</p>
<ul>
<li>Emmy*: Granada brings faster finality to Tezos. With the <a href="https://research-development.nomadic-labs.com/emmy-seven-years-of-updatable-consensus.html">Emmy*</a> update of the consensus algorithm, the time between blocks can now be as low as 30 seconds. Also the number of endorsement slots per block has been increased from 32 to 256. Other constants including rewards and security deposits have been updated.</li>
<li>Liquidity Baking: A decentralized exchange between tez and wrapped bitcoins (via tzBTC) will be deployed during the activation including a mechanism to incentivize decentralized provision of liquidity.</li>
<li>Gas Improvements: a refactoring of the Michelson interpreter has led to significant <a href="https://research-development.nomadic-labs.com/a-tale-of-two-reductions-in-gas-consumption-in-tezos.html">performance improvements</a> (typically a decrease of three to six times for already deployed contracts). Note that unfortunately this refactor has introduced a <a href="https://research-development.nomadic-labs.com/granada-comparisons-bug.html">non-critical bug</a> for smart contracts using the <code>COMPARE</code> operator.</li>
</ul>
<p>Congratulations to everyone involved in the development of this amendment and welcome to the Tezos blockchain, Granada!</p>
<p>We intend to inject our next proposal “H” before the end of September. It has yet more interesting features, large and small. Stay tuned, and happy baking!</p>Follow-up on the verification of Liquidity Baking smart contracts2021-08-03T18:00:00+02:002021-08-03T18:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-08-03:/follow-up-on-the-verification-of-liquidity-baking-smart-contracts.html<p>Granada has passed all three stages of voting and will be activated at the end of the Adoption period, which would likely take place this week on August 6. Liquidity Baking is one of the key features of Granada. We already wrote a <a href="https://research-development.nomadic-labs.com/progress-report-on-the-verification-of-liquidity-baking-smart-contracts.html">first progress report on the verification of Liquidity Baking smart contracts</a> in which we have provided an in-depth description of our effort to provide strong safety and correctness assurances. In this post, we would like to give a short update on our verification efforts around Liquidity Baking smart contracts.</p>
<p>In a nutshell, these efforts can be divided into two approaches: (1) formal verification of the <span class="caps">CPMM</span> contract using the Mi-Cho-Coq framework, and (2) intensive integration testing using property-based testing frameworks. Most notably, the tests implemented in (2) were focusing on the hypotheses used to conduct our formal proofs in (1).
Recently, we have decided to increase the scope of our integration tests to also include some key properties proven in Coq using Mi-Cho-Coq. In particular, we are now testing that:</p>
<ul>
<li>All three balances of the <span class="caps">CPMM</span> contract are always strictly positive</li>
<li>The <span class="caps">CPMM</span> invariant (see its definition <a href="https://research-development.nomadic-labs.com/progress-report-on-the-verification-of-liquidity-baking-smart-contracts.html#evolution-of-the-product-of-supplies">here</a>) is strictly increasing.</li>
</ul>
<p>Having in our integration tests properties that have been proven in Coq may seem redundant but these tests will act as regression tests in case the <span class="caps">CPMM</span> contract is ever changed in future protocol update. Furthermore, it is reassuring to have results validated by two different methods.</p>
<h2 id="other-audits-and-verification-efforts"><a class="toclink" href="#other-audits-and-verification-efforts">Other audits and verification efforts</a></h2>
<p>Speaking of different methods, we would also like to highlight that audits and verification efforts for Liquidity Baking smart contracts have been made outside of Nomadic.</p>
<p>Indeed an <a href="https://leastauthority.com/static/publications/LeastAuthority-Tezos-TzBTC-Final-Audit-Report.pdf">audit of tzBTC</a> has been published in March 2020 by Least Authority.</p>
<p>Furthermore <a href="https://runtimeverification.com/">Runtime Verification</a> have published a <a href="https://raw.githubusercontent.com/runtimeverification/publications/main/reports/smart-contracts/Tezos-Dexter.pdf">Formal Verification Report on Dexter 2</a> and have just released <a href="https://raw.githubusercontent.com/runtimeverification/publications/main/reports/smart-contracts/Tezos-Dexter-Liquidity-Baking.pdf">one on Liquidity Baking</a>, which was based on the same contract.</p>
<p>Runtime Verification’s verification approach shares some similarities with our formal verification efforts with Mi-Cho-Coq. In both cases, the work relies on a formal semantics of Michelson (<a href="https://runtimeverification.com/blog/k-michelson-a-case-study-on-formal-executable-language-specification-part-1">K-Michelson</a> for Runtime Verification, Mi-Cho-Coq for Nomadic Labs) that permits to rigorously reason about the behavior of programs. Note also that similar properties have been proven (<em>e.g.</em>, the functional correctness of the contracts’ Michelson bytecode, or the fact that the <span class="caps">CPMM</span> invariant is strictly increasing).</p>
<p>Of course, our approaches, while similar, differ in some aspects. In particular, Mi-Cho-Coq is based on <a href="https://coq.inria.fr">Coq</a>, while K-Michelson is based on the <a href="https://kframework.org/">K Semantic Framework</a>. We can also notice that the verification effort of Runtime Verification also covers the <span class="caps">FA1</span>.2 contract used by the <span class="caps">CPMM</span> to manage the tokens distributed to the liquidity providers. Overall, we believe that our approaches are complementary.</p>
<p>We are glad that several teams have independently studied the security of the Liquidity Baking contracts, as it gives the community more assurances that Liquidity Baking is safe to use.</p>Three questions to Nomadic Labs interns — Étienne Marais2021-08-02T11:00:00+02:002021-08-02T11:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-08-02:/three-questions-to-nomadic-labs-interns-etienne-marais.html<p>Short interview with Étienne Marais</p><p>Each year we host around half-a-dozen interns (<em>stagiaires</em>), for periods of two to six months.
These young men and women are usually studying or have just finished studying for undergraduate or Master’s degrees, and are eager to gain experience in the blockchain industry.
Each has a unique story to tell about their individual interest in this technology.</p>
<p>In this blogpost, we will ask three questions of one of our current interns: <strong>Étienne Marais</strong> (and a couple of questions of his mentors at Nomadic Labs).</p>
<p>Étienne — so happy to have you with us!
We hope you’re having a wonderful and educational period with Nomadic Labs.
We’d love to hear a bit about you and your activities <em>chez nous</em> …</p>
<h1 id="questions-for-etienne"><a class="toclink" href="#questions-for-etienne">Questions for Étienne</a></h1>
<h2 id="1-please-present-yourself-and-your-academic-background"><a class="toclink" href="#1-please-present-yourself-and-your-academic-background">1. Please present yourself and your academic background</a></h2>
<p>My name is Étienne Marais, I’m 24 years old, and I live in Paris.
I’ve been an OCaml enthusiast for 4 years now!
I’m really interested in the free software movement and the open source community.
I studied undergraduate computer science at Paris Diderot University from September 2016 to August 2019.
Then, I started a two-year master’s degree at <a href="https://u-paris.fr/">Université de Paris</a> (former Paris Diderot University) in September 2019, and I’m now in my last year.
My studies are mostly on programming languages, especially functional programming and compilation.</p>
<p>I got my interest in OCaml and blockchains from a pair of summer internships: two months in the summer of 2018 with <span class="caps">IRIF</span> (Research Institute on Fundamental Computer Science) showed me OCaml; then two months in the summer of 2020 with the OCaml Software Foundation working on the Learn-OCaml platform brought me into contact with the Tezos ecosystem and Nomadic Labs.
I learned about blockchains and the possibilities they offer for decentralization, self-governance and transparency. </p>
<p>I’m concerned about global warming and ecology, so how blockchains deal with their energy consumption is an important issue to me.
To answer it, I’ve dug further into the subject and learned about the difference between Proof of Work (which by design consumes a lot of energy) and the Proof of Stake (which consumes less energy, and is what is used in Tezos).</p>
<h2 id="2-tell-us-more-about-your-internship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months"><a class="toclink" href="#2-tell-us-more-about-your-internship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months">2. Tell us more about your internship: main subject, who is your mentor, what you have learned and especially, what surprised you the most within these months?</a></h2>
<p>In March 2021, I started an internship with Victor Allombert, Julien Tesson and Mathias Bourgoin on the energy consumption of Tezos.
There are two goals:</p>
<ol>
<li>To provide the Tezos community with tools to monitor the energy consumption of their live Tezos nodes. </li>
<li>To develop a benchmarking tool which will allow developers to profile the energy consumption of a Tezos node while performing various standard functions in the lab (i.e. under controlled and replicable conditions), such as baking a block or executing a simple smart contract.</li>
</ol>
<p>This internship is really challenging and stimulating, for two reasons: </p>
<ul>
<li>
<p>Firstly, it has pushed me deeper in the OCaml world, and it’s really thrilling to see the capabilities of the language in such a big project.
Discussing with other developers about how the tools I’m writing could be integrated into the actual workflow is amazing, because they always have good suggestions and help me improve the quality of my work.</p>
</li>
<li>
<p>Secondly, profiling computer performance is not new, but optimising the energy-efficiency of computer systems (some call this <a href="https://en.wikipedia.org/wiki/Green_computing"><strong>green computing</strong>, or <strong><span class="caps">ICT</span> sustainability</strong></a>) is a relatively recent field of research.
There are few precedents for this and we have to build the tools we need from scratch.
It’s really stimulating to work with my mentors to try to produce a tool which is as accurate as possible.</p>
</li>
</ul>
<p>I’m amazed by the technologies used to make a blockchain work, and how every person on the project is important to make the project succeed.
It is exciting to be part of such a big project and make contributions to it.</p>
<h2 id="3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-internship"><a class="toclink" href="#3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-internship">3. Why did you choose to become a part of the blockchain ecosystem and Nomadic labs? What are your plans after completing this internship?</a></h2>
<p>I decided to join Nomadic Labs because people working here are close to the research world.
Many of them have a PhD, so they are used to sharing their knowledge with others, and this creates a healthy place to learn the skills to develop a blockchain ecosystem.
Furthermore, they belong to the OCaml community, so I get to learn about the design of those libraries in the ecosystem that they maintain.</p>
<p>I wouldn’t have called myself a blockchain enthusiast before starting my internship with Nomadic Labs, but now I’ve discovered that blockchain is a technology that exists at the intersection of many important fields, including formal verification, networking, and concurrency — and with broad applications that are not limited to finance.</p>
<p>After the internship, I will leave Nomadic Labs as I want to discover other companies, but I intend to join another company working on Tezos, and to continue helping develop the OCaml ecosystem.</p>
<h1 id="questions-for-etiennes-nomadic-labs-mentors-victor-allombert-and-mathias-bourgoin"><a class="toclink" href="#questions-for-etiennes-nomadic-labs-mentors-victor-allombert-and-mathias-bourgoin">Questions for Étienne’s Nomadic Labs mentors Victor Allombert and Mathias Bourgoin</a></h1>
<h2 id="what-is-your-input-to-the-work-of-etienne"><a class="toclink" href="#what-is-your-input-to-the-work-of-etienne">What is your input to the work of Étienne?</a></h2>
<p>Through regular meetings and spontaneous questions, we discuss, exchange and propose various solutions and ideas to solve our problems. This guides Étienne toward the implementation of prototypes that we validate and improve together over time, leading to viable implementations.</p>
<h2 id="why-is-the-topic-of-this-internship-important"><a class="toclink" href="#why-is-the-topic-of-this-internship-important">Why is the topic of this internship important?</a></h2>
<p>There’s no point in building a technology nowadays unless it’s green.
Unlike Proof-of-Work blockchains, Tezos’ Proof-of-Stake inherently requires much less energy and cost to operate, but we need to back this up and optimize performance using concrete figures.
Fine-grained performance monitoring is necessary to understand the energy consumption of a node, and thus of the network overall.
Thanks to such tools and metrics, we can continuously reduce the ecological (and economic) footprint of Tezos.</p>Emmy: seven years of updatable consensus2021-07-30T12:00:00+02:002021-07-30T12:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-07-30:/emmy-seven-years-of-updatable-consensus.html<p>Survey of Emmy in its three flavours</p><p>Tezos is a blockchain<sup id="fnref:one-true"><a class="footnote-ref" href="#fn:one-true">1</a></sup>, and a blockchain is a distributed database that gets updated by adding <em>blocks</em> (small bunches of database operations).
Tezos’ basic design goes back to <a href="http://web.archive.org/web/20210216201223/https://tezos.com/static/white_paper-2dc8c02267a8fb86bd67a108199441bf.pdf">a 2014 whitepaper</a>, which was genuinely groundbreaking<sup id="fnref:no-hyperbole"><a class="footnote-ref" href="#fn:no-hyperbole">2</a></sup> — not least in its insight of making the protocol of the database itself be a mutable database entry; this is the so-called <a href="https://tezos.gitlab.io/alpha/voting.html">Tezos protocol upgrades</a>.
From this idea has grown a <a href="https://tezos.foundation/tezos-2020-year-in-review/">broad blockchain ecosystem</a>.</p>
<p>A key design choice is the <strong>consensus algorithm</strong>: how our system decides which blocks to add, and which to discard.
Thus far, Tezos has used and refined a Nakamoto-style consensus algorithm called <strong>Emmy</strong>.
Consensus algorithms are genuinely subtle,<sup id="fnref:subtle"><a class="footnote-ref" href="#fn:subtle">3</a></sup>
so it is high time that we explain what Emmy is and how it works:</p>
<ul>
<li><a href="#a-survey-and-explanation-of-emmy">A survey (and explanation) of Emmy</a></li>
<li><a href="#how-does-the-emmy-family-work">How does the Emmy family work?</a><ul>
<li><a href="#the-rules">The rules</a></li>
<li><a href="#high-level-view-">High-level view …</a></li>
<li><a href="#-in-more-detail">… in more detail</a></li>
<li><a href="#why-we-need-a-consensus-algorithm">Why we need a consensus algorithm</a></li>
<li><a href="#large-fast-networks-beat-small-slow-ones">Large fast networks beat small slow ones</a></li>
<li><a href="#endorsements">Endorsements</a></li>
</ul>
</li>
<li><a href="#forks">Forks</a><ul>
<li><a href="#the-rules-the-emmy-family-from-the-point-of-view-of-a-participant">The rules: the Emmy family, from the point of view of a participant</a></li>
<li><a href="#the-need-for-a-chain-fitness-function">The need for a chain fitness function</a></li>
<li><a href="#reasons-for-the-change-in-fitness-function">Reasons for the change in fitness function</a></li>
</ul>
</li>
<li><a href="#the-three-flavours-of-emmy">The three flavours of Emmy</a><ul>
<li><a href="#list-of-flavours">List of flavours</a></li>
<li><a href="#the-delay-function-of-emmy">The Delay function of Emmy</a></li>
<li><a href="#the-delay-function-of-emmy-1">The Delay function of Emmy+</a></li>
<li><a href="#the-delay-function-of-emmy-2">The Delay function of Emmy*</a></li>
</ul>
</li>
<li><a href="#the-mathematics-of-withstanding-attacks">The mathematics of withstanding attacks</a><ul>
<li><a href="#an-attack-scenario-a-valuable-car">An attack scenario (a Valuable Car)</a></li>
<li><a href="#calculating-the-confirmation-number">Calculating the confirmation number</a></li>
<li><a href="#setting-the-scene-and-some-simplifying-assumptions">Setting the scene, and some simplifying assumptions</a></li>
<li><a href="#assume-that-everyone-favours-themselves-and-the-network-favours-the-attacker">Assume that everyone favours themselves, and the network favours the attacker</a></li>
<li><a href="#some-probabilities">Some probabilities</a></li>
</ul>
</li>
<li><a href="#forks-starting-now">Forks starting now</a></li>
<li><a href="#forks-started-in-the-past">Forks started in the past</a></li>
<li><a href="#a-tool-and-the-attack-model">A tool, and the attack model</a><ul>
<li><a href="#a-concrete-tool">A concrete tool</a></li>
<li><a href="#notes-on-the-attack-model">Notes on the attack model</a></li>
</ul>
</li>
<li><a href="#the-future">The future</a><ul>
<li><a href="#further-reading">Further reading:</a></li>
</ul>
</li>
</ul>
<p><span class="math">\(\newcommand\ie{\mathit{ie}} \newcommand\dpp{\mathit{pd}}
\newcommand\te{\mathit{te}} \newcommand\md{\mathit{md}}
\newcommand\db{\mathit{bd}} \newcommand\de{\mathit{ed}}
\newcommand\trz{\mathit{level\_reward\_zero}}
\newcommand\tr{\mathit{level\_reward\_nonzero}}
\newcommand\br{\mathit{block\_reward}}
\newcommand\er{\mathit{endorsement\_reward}} \newcommand\f[1]{\mathit{#1}}
\newcommand{\set}[1]{\{#1\}} \newcommand{\var}[1]{\f{#1}}
\newcommand\epdelay{\mathit{emmy}^{\text{+}}\hspace{-.12cm}\_\mathit{delay}}
\newcommand\esdelay{\mathit{emmy}^{\star}\hspace{-.12cm}\_\mathit{delay}}
\newcommand\edelay{\f{delay}}
\newcommand\edelayp{\f{delay^+}}
\newcommand\edelays{\f{delay^*}} % syntax highlighting hack
\newcommand\edelaydelta{\edelay_{\Delta}} \newcommand\edelays{\f{delay^\star}}
\newcommand\params{\lambda} \newcommand\ie{\f{ie}} \newcommand\te{\f{te}}
\newcommand\dpp{\f{dp}} \newcommand\de{\f{de}} \newcommand\dpde{\f{dpde}}
\newcommand\bd{\f{bd}} \newcommand\bdz{\f{bd_0}} \newcommand\od{\tau}
\newcommand\emmy{\f{Emmy}} \newcommand\emmyp{\f{Emmy^+}}
\newcommand\emmys{\f{Emmy^{\star}}} \newcommand\emmyf{[\f{Emmy}]}
\newcommand\hb{H} \newcommand\cb{D} \newcommand\pr{\mathtt{Pr}}
\newcommand\pprio[1]{\mathbb{P}_{prio}(#1)}
\newcommand\pendo[1]{\mathbb{P}_{endo}(#1)}
\newcommand\pdiff[1]{\mathbb{P}_{\Delta}(#1)} \newcommand\secu{\eta}
\newcommand\steal{\f{sb}} \newcommand\seen{\f{seen}}
\newcommand\pseen[1]{\mathbb{P}^{#1}} \newcommand\psteal[1]{\mathbb{P}^{#1}}
\newcommand\pcst{\f{no\_delay}} \newcommand\pcstn{\f{past}}
\newcommand\pc{p^D} \newcommand\maxp{\f{max\_p}} \newcommand\ph{p^H}
\newcommand\barpc{\bar{p}^D} \newcommand\barph{\bar{p}^H} \newcommand\tsh{t^H}
\newcommand\tsc{t} \newcommand\bh{b^H}
\newcommand\difff[1]{\f{diff\_first}(#1)}
\newcommand\diff[1]{\f{diff\_subseq}(#1)}
\newcommand\diffl[1]{\f{diff}_{\ell}(#1)}
\newcommand\difffp[1]{\f{diff\_first^{\f{no\_delay}}}(#1)}
\newcommand\diffp[1]{\f{diff\_subseq^{\f{no\_delay}}}(#1)}
\newcommand\diffg{\f{diff}}
\newcommand\diffgns{\f{diff}^{ns}} \newcommand\diffgs{\f{diff}^{s}}
\newcommand\cchain{\f{ch}^\cb} \newcommand\hchain{\f{ch}^\hb}\)</span></p>
<h1 id="a-survey-and-explanation-of-emmy"><a class="toclink" href="#a-survey-and-explanation-of-emmy">A survey (and explanation) of Emmy</a></h1>
<p>In this blog post we will give an explanation and survey of the Emmy family of consensus algorithms, which has powered the Tezos blockchain since its inception.
We will start with</p>
<ul>
<li><a href="#how-does-it-work">an explanation of how the Emmy family works</a>, then</li>
<li><a href="#withstanding-attacks">discuss in specific detail the mathematics of withstanding attacks</a>, then</li>
<li><a href="#forks-starting-now">discuss “forks starting now”</a>, then</li>
<li><a href="#forks-starting-in-the-past">discuss “forks starting in the past”</a>, then</li>
<li><a href="#tool">mention a concrete tool to compute expected confirmation numbers</a> — how long you expect to wait for transactions to become final; bankers would call this the “settlement time” — and finally</li>
<li><a href="#future">we discuss the future</a> — including a planned move to <a href="https://arxiv.org/abs/2001.11965">Tenderbake</a> (which has also been the topic of <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">a previous blog post</a>).</li>
</ul>
<p>We mentioned there are several flavours of Emmy:</p>
<ul>
<li>Currently, Tezos uses the <a href="https://research-development.nomadic-labs.com/emmy-an-improved-consensus-algorithm.html">Emmy+</a> consensus algorithm.</li>
<li>This is a successor<sup id="fnref:name"><a class="footnote-ref" href="#fn:name">4</a></sup> to <a href="http://web.archive.org/web/20210216201223/https://tezos.com/static/white_paper-2dc8c02267a8fb86bd67a108199441bf.pdf">Emmy</a>.</li>
<li>An upgrade to <a href="https://gitlab.com/tezos/tzip/-/blob/master/drafts/current/draft_emmy-star.md">Emmy*</a> is imminent in the <a href="https://forum.tezosagora.org/t/announcing-granada/3183">Granada protocol proposal</a> (<a href="https://tzstats.com/election/29">election</a>).</li>
</ul>
<p>So ‘Emmy’ is a family of consensus algorithms —
technically, a collection of <em>proof-of-stake Nakamoto-style consensus algorithms</em>
— parameterised over some dials and switches which we can be tweaked to optimise safety and performance — and ideally, both at the same time.</p>
<p>These parameters include (explanations will follow below):
<a name="list"></a></p>
<ol>
<li>the <a href="#endorsement-slots">number of endorsement slots per block</a>,</li>
<li>the <a href="#delay">minimal delay function</a>, and</li>
<li>the <a href="#fitness">chain fitness function</a>.</li>
</ol>
<p>The evolution from
<a href="http://web.archive.org/web/20210216201223/https://tezos.com/static/white_paper-2dc8c02267a8fb86bd67a108199441bf.pdf">Emmy</a> to
<a href="https://research-development.nomadic-labs.com/emmy-an-improved-consensus-algorithm.html">Emmy+</a> to
<a href="https://gitlab.com/tezos/tzip/-/blob/master/drafts/current/draft_emmy-star.md">Emmy*</a>
consists of tweaks to these parameters, each of which had to be carefully considered and tested.
These tweaks are <em>practically</em> motivated: each iteration offers worthwhile real-world improvements in either speed or security or (preferably) both.<sup id="fnref:emmystarsecure"><a class="footnote-ref" href="#fn:emmystarsecure">5</a></sup></p>
<p>The evolution of these algorithms has been a part of the evolution of Tezos itself, and it illustrates just how far it is possible to optimise Nakamoto-style consensus in a live, functioning, industrial blockchain.
So …</p>
<p><a name="how-does-it-work"></a></p>
<h1 id="how-does-the-emmy-family-work"><a class="toclink" href="#how-does-the-emmy-family-work">How does the Emmy family work?</a></h1>
<p>A blockchain is a sequence of blocks.
We call position of a block in this sequence its <strong>level</strong>:
thus, the “block at level <span class="math">\(l\)</span>” is the <span class="math">\((l{+}1)\)</span>-th block in the sequence.
We start counting at zero, so the first block (also called the <strong>genesis</strong> block) is at level <span class="math">\(0\)</span>.
(You can view <a href="https://tzstats.com/0">details of the genesis block of the Tezos blockchain</a>.)</p>
<p>Blocks are timestamped.
So the block at level <span class="math">\(l\)</span> will also have some timestamp <span class="math">\(t_l\)</span>.</p>
<p><a name="agreement"></a>
For simplicity suppose also that the blockchain is <strong>unforked</strong>, meaning that the network agrees on the blockchain up to and including block <span class="math">\(B\)</span> at level <span class="math">\(l\)</span> (we consider the more general case of a forked chain <a href="#no-agreement">below</a>).</p>
<p>Now it’s time to decide on what block to add at level <span class="math">\(l{+}1\)</span> to our unforked chain.
The consensus algorithm gets to work, according to the following rules:</p>
<h3 id="the-rules"><a class="toclink" href="#the-rules">The rules</a></h3>
<h4 id="high-level-view"><a class="toclink" href="#high-level-view">High-level view …</a></h4>
<p>At a high level, Emmy acts as follows:
<a name="rules"></a></p>
<ol>
<li>sort participants into random order,</li>
<li>query each participant in order for a block, and</li>
<li>add the first valid block received, to the blockchain.</li>
</ol>
<p>But this is a <em>blockchain system</em>, so we need an algorithm that can achieve the effect above and</p>
<ul>
<li>the algorithm is scalable, efficient, and resilient to real-world faults such as network interruptions, and</li>
<li>does not assume a central authority, and</li>
<li>does not assume all participants are well-behaved.</li>
</ul>
<h4 id="in-more-detail"><a class="toclink" href="#in-more-detail">… in more detail</a></h4>
<p>We will need two inputs to the algorithm:</p>
<p><a name="random_seed"></a></p>
<ol>
<li>
<p>We need a <strong>random seed</strong>.</p>
<p>This is a number used to <em>seed</em> a (pseudo)random generating function.
This is obtained as a hash of the state of the blockchain approximately <span class="math">\(4096\times 5\)</span> blocks in the past — at one block per minute that means <span class="math">\(20480\)</span> minutes ago (about two weeks).</p>
</li>
<li>
<p>We need a mathematical function called the <strong>delay function</strong>.</p>
<p><span class="math">\(\mathit{delay}\)</span> has type <span class="math">\(\mathbb N\times\mathbb N\to\mathbb N₊\)</span>, meaning that it inputs two nonnegative numbers and returns a strictly positive number.
<span class="math">\(\mathit{delay}\)</span> is a parameter of the Emmy algorithm, meaning that it varies between Emmy, Emmy+, and Emmy*; for now, it just suffices that this exists.
More on this shortly.</p>
</li>
</ol>
<p>We now proceed as follows:</p>
<ol>
<li>
<p>Using the <a href="#random_seed">random seed</a> described above, each <a href="https://tezos.gitlab.io/008/proof_of_stake.html/#active-and-passive-delegates">active blockchain participant</a> <span class="math">\(a\)</span> with at least one roll ( = 8,000 ꜩ) is randomly assigned two pieces of data:</p>
<ul>
<li>a unique <strong>earliest priority</strong> <span class="math">\(p(a)\)</span>, which is a non-negative number.<sup id="fnref:how-allocated"><a class="footnote-ref" href="#fn:how-allocated">7</a></sup> Smaller numbers are better and each participant gets a different number (think: positions in a queue).</li>
<li>an <strong>endorsement power</strong> <span class="math">\(w(a)\)</span>, which is a number between 0 and 32 (think: magic talismans; see <a href="#talisman">below</a>). <a name="endorsement-power"></a></li>
</ul>
<p>Note that the earliest priority and the endorsement power are abstract data quantities.
We discuss below how <a href="#delay-function-equation">these quantities get mapped to a time in seconds</a>.</p>
<p>The distribution of earliest priority <span class="math">\(p(a)\)</span> is proportional to participants’ stake: if <span class="math">\(a\)</span> has more stake then there is a correspondingly increased chance that e.g. <span class="math">\(p(a)=0\)</span> or <span class="math">\(p(a)=1\)</span>.</p>
<p>Equivalently: priorities are dispensed by a uniform random distribution on a per-roll basis, so a participant who controls more rolls, will get proportionally more priorities.</p>
<p>The distribution of endorsement power <span class="math">\(w(a)\)</span> is also proportional to participants’ stake, and furthermore the total endorsement power dispensed must sum to 32. Equivalently: the 32 endorsement slots are distributed per roll, so a participant who controls more rolls, is proportionally more likely to get one or more endorsement slots.</p>
<p>So intuitively: <a name="talisman"></a></p>
<blockquote>
<p><em>imagine that Tezos is a fantasy magic kingdom which dispenses queue positions and a strictly limited total of 32 magic talismans to its landowners — and, because this is a proof-of-stake system, larger landowners tend to get better queue numbers and more talismans. We will continue this example <a href="#talisman-2">below</a>.</em></p>
</blockquote>
</li>
<li>
<p>Recall that <a href="#agreement">we assumed for simplicity</a> that all the participants agree on the state of the blockchain so far, and in particular that it ends at block <span class="math">\(B\)</span> at level <span class="math">\(l\)</span>.</p>
<p>Now, any participant <span class="math">\(a\)</span> can <strong>endorse</strong> (vote for) <span class="math">\(B\)</span> by transmitting an endorsement operation <span class="math">\(d(a,B)\)</span> across the network.</p>
<p>So suppose at some particular moment that block <span class="math">\(B\)</span> has received a set <span class="math">\(e_B=\{d(a_1,B), d(a_2,B), \dots, d(a_n,B)\}\)</span> of endorsements from distinct participants <span class="math">\((a_1,a_2,\dots,a_n)\)</span> with endorsement powers <span class="math">\((w(a_1),w(a_2),\dots,w(a_n))\)</span> respectively.<sup id="fnref:distributed"><a class="footnote-ref" href="#fn:distributed">8</a></sup>
Write <span class="math">\(w(e_B)\)</span> for the sum of the endorsement powers of <span class="math">\(a_1\)</span> to <span class="math">\(a_n\)</span>:
<div class="math">$$
w(e_B) = w(a_1) + w(a_2) + \dots + w(a_n) .
$$</div>
</p>
<p>We see that each endorsement is weighted by the endorsement power <span class="math">\(w(a_i)\)</span> of <span class="math">\(a_i\)</span> — so in effect, at most 32 participants can usefully endorse <span class="math">\(B\)</span>, since there are at most 32 endorsement slots to go around.</p>
<p><a name="delay-function-equation"></a> Then <strong>a participant with earliest priority <span class="math">\(p(a)\)</span> has the right to bake a valid block <span class="math">\(B'\)</span> and attach it to a chain after a delay of <div class="math">$$ delay(p(a),w(e_B)) $$</div> seconds</strong>. <a href="#delay">We discuss in detail below what function <span class="math">\(delay\)</span> is</a>; for now the function is just a parameter of the algorithm.</p>
<p>Every block contains enough information (timestamp, priority, public key of creating participant, endorsements, and so on) that every participant can check — just by examining it and without reference to any external oracle — whether it validly follows on from the last block.<sup id="fnref:time"><a class="footnote-ref" href="#fn:time">6</a></sup></p>
<p>Then <span class="math">\(B'\)</span> is transmitted to the network and, assuming everything is working efficiently, <span class="math">\(B'\)</span> gets appended to the blockchain at level <span class="math">\(l{+}1\)</span>.</p>
</li>
<li>
<p>If everything is not working efficiently — perhaps the network is slow or participant <span class="math">\(a\)</span> is rebooting — then time passes and eventually a delay of <span class="math">\(delay(p(a'),w(e_B'))\)</span> may pass, where <span class="math">\(a'\)</span> is the next participant in line and <span class="math">\(e_B'\)</span> is the number of endorsements that <span class="math">\(a'\)</span> sees for <span class="math">\(B\)</span>. From this moment, both <span class="math">\(a\)</span> and <span class="math">\(a'\)</span> can bake blocks. In particular, if <span class="math">\(a\)</span> remains <em>hors combat</em> then <span class="math">\(a'\)</span> may be the one to bake the next valid block. And so it goes on until somebody bakes a valid block and the blockchain is extended.</p>
</li>
</ol>
<p>It might be helpful to continue our analogy of Tezos as a magic kingdom:</p>
<p><a name="talisman-2"></a></p>
<blockquote>
<p><em>Suppose the Tezos blockchain is a magic kingdom and suppose block <span class="math">\(B\)</span> at level <span class="math">\(l\)</span> is the most recent Law of the magic kingdom to be passed.
Landowners line up in a random order and 32 magic talismans are randomly dispensed, where both queue order and talisman distribution are weighted according to how much land each landowner owns.
Landowners use their talismans to broadcast a magical blessing of the most recent Law, and — depending on the delay function, which depends on a landowner’s queue position, and on the talisman-weighted number of blessings of the most recent Law <span class="math">\(B\)</span> that the landowner has received — the first landowner to bake a block, gets to determine the next <span class="math">\(B'\)</span>, i.e. to write the next law of the magic kingdom. Of course, this means that large landowners connected to large sections of the magical blessing broadcast network, tend to get early queue positions and receive lots of the magical blessings and so tend to set the laws; whereas those with smaller parcels of land or more restricted access to the network, tend to miss out.</em></p>
</blockquote>
<h4 id="why-we-need-a-consensus-algorithm"><a class="toclink" href="#why-we-need-a-consensus-algorithm">Why we need a consensus algorithm</a></h4>
<p>If this seems complicated, note what it buys us:</p>
<ul>
<li>there is no central authority, and</li>
<li>the system self-organises to reward having a large stake and following the rules, and is resilient against attack, non-participation, and network delays.</li>
</ul>
<p><a name="no-agreement"></a></p>
<h3 id="large-fast-networks-beat-small-slow-ones"><a class="toclink" href="#large-fast-networks-beat-small-slow-ones">Large fast networks beat small slow ones</a></h3>
<p>The algorithm dispenses endorsements amongst all active participants.
Thus if a small group of nodes becomes isolated for a while — or just decides to go its own way — then it is <em>not</em> the case that they can just give one another early priorities, vigorously endorse one another’s blocks, and so build a long chain.<sup id="fnref:academia"><a class="footnote-ref" href="#fn:academia">9</a></sup></p>
<p>An isolated group may fork, but</p>
<ul>
<li>its members can garner relatively few priorities (because statistically speaking, most priorities go to participants not in the group) and</li>
<li>relatively few endorsement slots (ditto),</li>
</ul>
<p>so that the values of the delay function within the group will be relatively large, and it will make relatively slow progress.
Thus, if and when the group tries to rejoin the main body of the network, the group’s branch will be short compared to the branch of the rest of the blockchain, and it will die off.</p>
<h3 id="endorsements"><a class="toclink" href="#endorsements">Endorsements</a></h3>
<p>At first sight it might seem strange to endorse the last block <span class="math">\(B\)</span>.
After all <span class="math">\(B\)</span> has already been attached to the blockchain, and we assumed for the sake of argument that there are no forks.
So why does it even matter if we now endorse it?</p>
<p>Endorsements tend to unify the network and kill off small, slow, isolated chains: a small fragment of the network will find it difficult to ignore the rest of the community and go its own way,</p>
<ul>
<li>not just because it will struggle to obtain small priorities, but also</li>
<li>because <em>even once it gets a priority</em> it may also struggle to gain endorsements from the wider network for the block that it bakes with that priority.</li>
</ul>
<p>In principle (withholding) endorsements could also be used to penalise <span class="math">\(B\)</span> if we disapprove of its contents.
In practice this is not done: participants just endorse <span class="math">\(B\)</span> as soon as they can (for which they are rewarded with endorsing rewards).
See <a href="#the-rules-the-emmy-family-from-the-point-of-view-of-a-participant">the algorithm from the point of view of a participant</a>.</p>
<h2 id="forks"><a class="toclink" href="#forks">Forks</a></h2>
<p>Recall that we <a href="#agreement">simplified</a> and assumed that all participants agree on the state of the blockchain so far.
But what if they don’t?
In practice the chain can <em>fork</em>, meaning that different participants have different views of the blockchain so far.<sup id="fnref:tenderbake-no-fork"><a class="footnote-ref" href="#fn:tenderbake-no-fork">10</a></sup></p>
<p>So what if the blockchain is forked?</p>
<p><a href="#list">Recall</a> that the consensus algorithm depends on these three parameters:</p>
<ul>
<li>the random seed, which was seeded from the state of the chain at least two weeks ago — we presume that no fork can last that long, so every participant now agrees on this quantity — and</li>
<li>the delay function, which is a fixed parameter of the current economic protocol, so again every participant agrees on this — and</li>
<li>the <strong>fitness function</strong> which is also a fixed parameter of the economic protocol.</li>
</ul>
<p>So we can assume that these parameters are shared and everybody is working from a shared notion of “what is a valid chain”.
Furthermore, checking validity of a proposed chain is precise and unambiguous: participants should just work from the <strong>fittest blockchain branch</strong> that they can see.
Note that the use of a fitness function to choose the canonical branch is common to Nakamato-style consensus algorithms.
This is known in the literature as the <em>fork choice rule</em>.</p>
<p>If the network becomes partitioned then the blockchain may fork for a while, and the partitions could evolve independently for a while, but it will snap back once connectivity is restored, provided that most (= at least half) of the participants work following <a href="#the-rules">the rules</a> above.
This is called a <em>Nakamoto-style</em> system because this is how Bitcoin works.<sup id="fnref:pos"><a class="footnote-ref" href="#fn:pos">11</a></sup></p>
<p><a name="the-rules"></a></p>
<h3 id="the-rules-the-emmy-family-from-the-point-of-view-of-a-participant"><a class="toclink" href="#the-rules-the-emmy-family-from-the-point-of-view-of-a-participant">The rules: the Emmy family, from the point of view of a participant</a></h3>
<ol>
<li>Continually observe blocks and endorsements on the network.</li>
<li>Work from the ‘best’ valid chain <span class="math">\(\mathcal B\)</span> that you observe. If the system is unforked then picking the best chain is easy because there is only one. If the system is forked, only switch to another chain if it is ‘better’. What ‘better’ means for a chain is determined by <a href="#fitness">a metric called <em>chain fitness</em>, discussed below</a>.</li>
<li>Endorse the first block you observe that can validly attach to <span class="math">\(\mathcal B\)</span>.</li>
<li>If you do not observe such a block, then produce one as soon as the delay function says that you validly can.</li>
</ol>
<p><a name="fitness"></a></p>
<h3 id="the-need-for-a-chain-fitness-function"><a class="toclink" href="#the-need-for-a-chain-fitness-function">The need for a chain fitness function</a></h3>
<p>Suppose we are on the blockchain and it is forked.
How is an honest participant to decide which chain is better?<sup id="fnref:economist"><a class="footnote-ref" href="#fn:economist">13</a></sup>
Here are two possible answers:</p>
<ol>
<li>
<p>In Emmy:</p>
<blockquote>
<p><strong>chain fitness</strong> is equal to the sum of the length of a chain and the number of endorsements which it contains.</p>
</blockquote>
<p>Thus, an honest participant who bakes on the fittest branch of a fork will prefer the branch with the most blocks-plus-endorsements.</p>
</li>
<li>
<p>In both Emmy+ and Emmy*:</p>
<blockquote>
<p><strong>chain fitness</strong> is equal to the length of the chain.</p>
</blockquote>
<p>Thus, an honest participant who bakes on the fittest branch of a fork will prefer the longest branch.<sup id="fnref:same-length"><a class="footnote-ref" href="#fn:same-length">14</a></sup></p>
</li>
</ol>
<h3 id="reasons-for-the-change-in-fitness-function"><a class="toclink" href="#reasons-for-the-change-in-fitness-function">Reasons for the change in fitness function</a></h3>
<p>This is <a href="https://research-development.nomadic-labs.com/emmy-an-improved-consensus-algorithm.html">discussed in the post on Emmy+:</a>
search for <em>“simplifies the optimal baking strategy”</em>.</p>
<p>The issue with the Emmy fitness function (item 1 in the list above) is that it gives equal weight to an endorsement and a block — and each block can hold up to 32 endorsements.
Thus, in terms of chain fitness, gathering endorsements for the last block is more important than producing the next block and thus extending the chain.</p>
<p>This incentivises bakers to hold off baking a block until they can pack it with as close as possible to a full complement of 32 endorsements, for fear that if they do not then their chain might be overtaken by a shorter fork (as much as ¹⁄₃₂ of the length) but with more endorsements and so greater fitness.</p>
<p>This incentive structure could slow down the blockchain overall, and in particular it could slow down transaction rate (the overall rate at which transactions get included in blocks and baked onto the chain), which is not good for users, who are likely to prefer high transaction rates and rapid transaction settlement times.</p>
<p>Emmy+ and Emmy* address this by setting chain fitness to equal chain length.
Endorsements still play a role, but the effect is exerted via the <a href="#delay">delay function</a> — discussed below; essentially, more endorsements means you can bake earlier.
Honest participants just prefer the longest chain and there is no reason not to bake a block, once the delay function permits it.</p>
<h2 id="the-three-flavours-of-emmy"><a class="toclink" href="#the-three-flavours-of-emmy">The three flavours of Emmy</a></h2>
<h3 id="list-of-flavours"><a class="toclink" href="#list-of-flavours">List of flavours</a></h3>
<p>Emmy exists in three flavours:</p>
<ol>
<li>The original version <a href="http://web.archive.org/web/20210216201223/https://tezos.com/static/white_paper-2dc8c02267a8fb86bd67a108199441bf.pdf">Emmy</a>. This was used from 2014 until <a href="https://tzstats.com/protocols/PsBabyM1eUXZseaJdmXFApDSBqj8YBfwELoxZHHW77EMcAbbwAS">18 October 2019</a> (this is the first block to use the Babylon protocol). Emmy is no longer current.</li>
<li>The new version <a href="https://research-development.nomadic-labs.com/emmy-an-improved-consensus-algorithm.html">Emmy+</a>. This is current.</li>
<li>The forthcoming upgrade to <a href="https://gitlab.com/tezos/tzip/-/blob/master/drafts/current/draft_emmy-star.md">Emmy*</a> in <a href="https://tzstats.com/election/29#adoption_period">Granada</a>.</li>
</ol>
<p>These are essentially the same algorithm but they differ in tweaks to their parameters. Specifically:</p>
<p><a name="endorsement-slots"></a></p>
<ul>
<li>Emmy and Emmy+ have 32 endorsement slots per level. Emmy* has 256, speeding up confirmation time and increasing user participation.<sup id="fnref:statistics"><a class="footnote-ref" href="#fn:statistics">15</a></sup></li>
<li>The chain fitness function of Emmy is “blocks + endorsements”; that of Emmy+ and Emmy* is just “blocks”, as <a href="#fitness">discussed above</a>.</li>
<li>The delay functions of the three algorithms differ, as <a href="#delay">discussed below</a>.</li>
</ul>
<p><a name="delay"></a></p>
<h3 id="the-delay-function-of-emmy"><a class="toclink" href="#the-delay-function-of-emmy">The Delay function of Emmy</a></h3>
<p>In Emmy, <span class="math">\(\edelay\)</span> is a function just of the priority <span class="math">\(p\)</span>:</p>
<div class="math">$$
\begin{equation}\tag{1}\label{eq:delay}
\begin{array}{r@{\ }l}
\edelay(p)
=& \db + \dpp \cdot p
\\
=& 60 + 75 \cdot p \quad \text{(seconds)}.
\end{array}
\end{equation}
$$</div>
<ul>
<li><span class="math">\(\db=60\)</span> is the <em>base delay</em>. Thus this is a constant minimal offset from one block to the next.</li>
<li><span class="math">\(\dpp=75\)</span> is the <em>priority delay</em>. Thus this establishes the time between priorities, as discussed above.</li>
</ul>
<p><strong>Worked examples:</strong></p>
<ul>
<li>A participant with earliest priority <span class="math">\(0\)</span> can start baking after <span class="math">\(60+75\cdot 0 = 60\)</span> seconds.</li>
<li>A participant with earliest priority <span class="math">\(1\)</span> can start baking after <span class="math">\(60+75\cdot 1 = 135\)</span> seconds.</li>
</ul>
<h3 id="the-delay-function-of-emmy_1"><a class="toclink" href="#the-delay-function-of-emmy_1">The Delay function of Emmy+</a></h3>
<p>Emmy+ adds a dependency on <span class="math">\(w\)</span>, the <a href="#endorsement-power">endorsement power</a> of the endorsements in the block to be baked:</p>
<div class="math">$$
\begin{equation}\tag{2}\label{eq:epdelay}
\begin{array}{r@{\ }l}
\edelayp(p, w)
=& \db + \dpp \cdot p + \de\cdot \max(0, \frac{3}{4}\cdot\te - w)
\\
=& 60 + 40 \cdot p + 8\cdot \max(0, 24 - w) \quad\text{(seconds)}.
\end{array}
\end{equation}
$$</div>
<p>It might help to view <span class="math">\(\mathit{delay}\)</span> abstractly as a function that tends to increase on the first argument (the priority slot), and decreases linearly on the second (the endorsement power). So:</p>
<ul>
<li>later priority slot = more delay;</li>
<li>more endorsements = less delay.</li>
</ul>
<p>In words, Equation \eqref{eq:epdelay} says that the delay is:</p>
<ul>
<li>a <em>base delay</em> of <span class="math">\(\db=60\)</span> seconds, plus</li>
<li>a <em>priority delay</em> of <span class="math">\(\dpp=40\)</span> seconds, plus</li>
<li>a <em>delay per missed endorsement</em> of <span class="math">\(\de=8\)</span> seconds for every endorsement slot that the block to be baked falls short of a threshold of <span class="math">\(24=\frac{3}{4}\cdot \te\)</span> out of the <span class="math">\(\te=32\)</span> available endorsement slots.</li>
</ul>
<p>From the <a href="/babylon-proposal-injected.html">Babylon protocol upgrade</a> (18 October 2019) to time of writing, these parameters have been fixed at
</p>
<div class="math">$$
\db = 60,\quad \dpp=40,\quad \de=8,\quad \te=32.
$$</div>
<p><strong>Worked example:</strong></p>
<ul>
<li>A participant with earliest priority <span class="math">\(0\)</span>, baking a block with <span class="math">\(16\)</span> endorsements for the previous block — thus, with half of the full complement of 32 possible endorsement slots — can start baking after <span class="math">\(60+40\cdot 0+8\cdot 8=124\)</span> seconds.</li>
<li>If the participant had gathered <span class="math">\(24\)</span> instead of <span class="math">\(16\)</span> endorsements, then this would drop to <span class="math">\(60\)</span> seconds.</li>
<li>A participant with earliest priority <span class="math">\(1\)</span>, baking a block with <span class="math">\(16\)</span> endorsements for the previous block — thus, with half of the full complement of 32 possible endorsement slots — can start baking after <span class="math">\(60+40\cdot 1+8\cdot 8=164\)</span> seconds.</li>
<li>If the participant had gathered <span class="math">\(24\)</span> instead of <span class="math">\(16\)</span> endorsements, then this would drop to <span class="math">\(100\)</span> seconds.</li>
</ul>
<p><a name="emmystar_delay"></a></p>
<h3 id="the-delay-function-of-emmy_2"><a class="toclink" href="#the-delay-function-of-emmy_2">The Delay function of Emmy*</a></h3>
<p><a href="https://gitlab.com/tezos/tzip/-/blob/master/drafts/current/draft_emmy-star.md">Emmy*</a> builds on the delay function of Emmy+ while observing that during normal operation most blocks are baked at priority 0 and with plenty of endorsements.
So let’s fast-track this optimal case:</p>
<div class="math">$$
\begin{equation}\tag{3}\label{eq:esdelay}
\edelays(p, w) =
\begin{cases}
\md & \text{ if } p = 0 \wedge w \geq \frac{3}{5}\te
\\
\edelayp(p, w) & \text{ otherwise}
\end{cases}
\end{equation}
$$</div>
<p>Above <span class="math">\(\md=30\)</span> is the minimal delay and <span class="math">\(\te=256\)</span> is the number of endorsement slots. Furthermore, the constant <span class="math">\(\de\)</span> used by <span class="math">\(\edelayp(p, w)\)</span> has been changed to 4.</p>
<p><strong>Worked example:</strong></p>
<ul>
<li>A participant with earliest priority <span class="math">\(0\)</span>, baking a block with <span class="math">\(153\)</span> endorsements for the previous block — thus, with less than 60% of the full complement of 256 possible endorsement slots — can start baking after <span class="math">\(60\)</span> seconds.</li>
<li>A participant with earliest priority <span class="math">\(0\)</span>, baking a block with <span class="math">\(154\)</span> endorsements for the previous block — thus, with at least 60% of the full complement of 256 possible endorsement slots — can start baking after <span class="math">\(30\)</span> seconds.</li>
<li>If the participant had gathered <span class="math">\(192\)</span> instead of <span class="math">\(155\)</span> endorsements, then this would make no difference and they can still start baking after <span class="math">\(30\)</span> seconds.</li>
<li>A participant with earliest priority <span class="math">\(1\)</span>, baking a block with <span class="math">\(128\)</span> endorsements for the previous block — thus, with half of the full complement of 256 possible endorsement slots — can start baking after <span class="math">\(60+40\cdot 1+4\cdot (192 - 128)=356\)</span> seconds.</li>
<li>If the participant had gathered <span class="math">\(192\)</span> instead of <span class="math">\(128\)</span> endorsements, then this would drop to <span class="math">\(60+40\cdot 1=100\)</span> seconds.</li>
</ul>
<p>So much for our overview of the delay function.
How does this help the blockchain to withstand attacks?
What does an attack look like, anyway?
Read on …</p>
<p><a name="withstanding-attacks"></a></p>
<h2 id="the-mathematics-of-withstanding-attacks"><a class="toclink" href="#the-mathematics-of-withstanding-attacks">The mathematics of withstanding attacks</a></h2>
<p><a name="valuable-car"></a></p>
<h3 id="an-attack-scenario-a-valuable-car"><a class="toclink" href="#an-attack-scenario-a-valuable-car">An attack scenario (a Valuable Car)</a></h3>
<p>Let the time be <span class="math">\(t=0\)</span> and consider the following scenario:</p>
<ol>
<li>You have a Valuable Car to sell.</li>
<li>I agree to purchase it and we agree on a sum of Quite A Lot for the sale.</li>
<li>I bake a block <span class="math">\(B\)</span> at time <span class="math">\(t=0\)</span> recording a transfer of Quite A Lot from my account to yours.</li>
<li>I also secretly bake another block <span class="math">\(B'\)</span> (this is called <em>double baking</em>). <span class="math">\(B'\)</span> is identical to <span class="math">\(B\)</span> except that this includes a transaction for Far Less from my account to yours.</li>
<li>The main Tezos chain <span class="math">\(\mathcal M\)</span> continues to evolve from <span class="math">\(B\)</span>, and (still in secret) I bake an alternative chain <span class="math">\(\mathcal S\)</span> evolving from <span class="math">\(B'\)</span> instead of from <span class="math">\(B\)</span>.</li>
<li>Meanwhile, I take possession of the Valuable Car, and drive it home.</li>
<li>When I get home, I reveal my chain to the network. In this chain I bought your car, but for Far Less rather than for Quite A Lot.</li>
</ol>
<p>If you complain, I can say “What’s the problem? I paid you and you gave me the Valuable Car”.</p>
<ul>
<li>Say that my attack <em>succeeds</em> when there exists some time <span class="math">\(t>0\)</span> at which <span class="math">\(\mathcal S\)</span> is longer than <span class="math">\(\mathcal M\)</span>. Then, I can reveal <span class="math">\(\mathcal S\)</span> to the world and as per <a href="#the-rules">the rules</a> honest participants will bake from <span class="math">\(\mathcal S\)</span>.</li>
<li>Say that my attack <em>fails</em> when for every time <span class="math">\(t>0\)</span>, <span class="math">\(\mathcal S\)</span> is shorter than <span class="math">\(\mathcal M\)</span>. I can still reveal <span class="math">\(\mathcal S\)</span> to the world, but as per <a href="#the-rules">the rules</a> it will be ignored.</li>
</ul>
<p>Whether or not this attack succeeds or fails depends on the relative rates of growth of <span class="math">\(\mathcal M\)</span> and <span class="math">\(\mathcal S\)</span>.
This is dictated by the <a href="#delay">delay function</a> as discussed above.
In the long term my chain <span class="math">\(\mathcal S\)</span> will be slower than <span class="math">\(\mathcal M\)</span> (so long as I have less than half the total stake on the chain) because I will gain relatively fewer priorities and endorsements.
Still: like a gambler in a casino, I might get lucky.</p>
<p>So we can ask:</p>
<ul>
<li>For a given attacker stake, what are the chances of an attacker undoing a transaction as outlined above?</li>
<li>Turning the question around: after how many blocks on the main chain <em>after</em> I paid you Quite A Lot can you feel safe about handing over your Valuable Car?</li>
</ul>
<h3 id="calculating-the-confirmation-number"><a class="toclink" href="#calculating-the-confirmation-number">Calculating the confirmation number</a></h3>
<h4 id="setting-the-scene-and-some-simplifying-assumptions"><a class="toclink" href="#setting-the-scene-and-some-simplifying-assumptions">Setting the scene, and some simplifying assumptions</a></h4>
<p>We can amalgamate multiple dishonest bakers <span class="math">\(\cb_i\)</span> into a single ‘composite’ dishonest baker <span class="math">\(\cb\)</span> having as stake fraction the sum of the stake fractions of <span class="math">\(\cb_i\)</span>, and similarly for the honest bakers.
We can also think of a fork as two competing chains — an honest <strong>main chain</strong> <span class="math">\(\mathcal M\)</span> and a dishonest <strong>hostile chain</strong> <span class="math">\(\mathcal S\)</span> — both extending some genesis block at level <span class="math">\(\ell = 0\)</span>.<sup id="fnref:ell"><a class="footnote-ref" href="#fn:ell">16</a></sup></p>
<p>Thus we reason henceforth using</p>
<ul>
<li>a single honest baker <span class="math">\(\hb\)</span> with stake fraction <span class="math">\(1-f\)</span> and</li>
<li>a single dishonest adversary baker <span class="math">\(\cb\)</span> with stake fraction <span class="math">\(f\)</span>,</li>
<li>both competing to add a longer chain starting from a common block at level <span class="math">\(\ell = 0\)</span>.</li>
</ul>
<p>We’re playing for the adversary <span class="math">\(\cb\)</span> in the mathematics below, so we write</p>
<ul>
<li><span class="math">\(f\)</span> for the stake fraction of <span class="math">\(\cb\)</span>,</li>
<li><span class="math">\(p\)</span> for the earliest priority of <span class="math">\(\cb\)</span>, and</li>
<li><span class="math">\(e\)</span> for the number of endorsement slots of <span class="math">\(\cb\)</span>.</li>
</ul>
<p>To win, the dishonest chain <span class="math">\(\mathcal S\)</span> of <span class="math">\(\cb\)</span> must accumulate more blocks than the honest main chain <span class="math">\(\mathcal M\)</span>.
So we need to compute the probability that the timestamp of the <span class="math">\(l\)</span>th and final block in <span class="math">\(\mathcal S\)</span> is <strong>less</strong> than the timestamp of the <span class="math">\(l\)</span>th and final block in <span class="math">\(\mathcal M\)</span>. If this happens, then <span class="math">\(\cb\)</span> can reveal the dishonest chain <span class="math">\(\mathcal S\)</span> to the network and participants will switch to it.</p>
<h4 id="assume-that-everyone-favours-themselves-and-the-network-favours-the-attacker"><a class="toclink" href="#assume-that-everyone-favours-themselves-and-the-network-favours-the-attacker">Assume that everyone favours themselves, and the network favours the attacker</a></h4>
<p><a name="delta"></a></p>
<p>We assume that:</p>
<ul>
<li>Everybody bakes as early as they can (thus handing minimal advantage to the adversary).</li>
<li>The honest baker’s messages take time <span class="math">\(\Delta\)</span> to arrive, which intuitively corresponds to some slowest possible path in the network.</li>
<li>The attacker’s messages arrive instantly.</li>
</ul>
<p>So this is a worst case scenario for the honest chain, in which honest network communications are as slow as they possibly could be, and dishonest attacker communications are very fast.
In symbols, we can calculate<sup id="fnref:calculate"><a class="footnote-ref" href="#fn:calculate">17</a></sup> that for a block baked by <span class="math">\(\hb\)</span>, the <strong>minimum delay function</strong> <span class="math">\(\edelaydelta(\ph, e)\)</span> is:
</p>
<div class="math">$$
\edelaydelta(\ph, e) = \min\bigl(\max(2\Delta, \edelay(\ph, e)),\ \max(\Delta, \edelay(\ph, 0))\bigr)
% \edelaydelta(p, e) = \edelay(p, e)$ for blocks baked by the
$$</div>
<h4 id="some-probabilities"><a class="toclink" href="#some-probabilities">Some probabilities</a></h4>
<p>We want to calculate the probability that the hostile chain <span class="math">\(\mathcal S\)</span> overtakes the main chain <span class="math">\(\mathcal M\)</span>, under the assumptions above.
This can be expressed in terms of differences between minimal block delays, using a large summation over the distribution of priorities and endorsements.</p>
<ul>
<li>Write <span class="math">\(\pprio{f,p}\)</span> for the probability that the earliest priority of a player with stake fraction <span class="math">\(f\)</span> is <span class="math">\(p\)</span>: <div class="math">$$\pprio{f,p} = \pr[best\_prio = p] = (1-f)^p f$$</div> (recall: the first priority is <span class="math">\(0\)</span>).</li>
<li>Write <span class="math">\(\pendo{f,e}\)</span> for the probability that <span class="math">\(\cb\)</span> gets <span class="math">\(e\)</span> many endorsement slots: <div class="math">$$\pendo{f,e}=\pr[num\_endo = e] = \textstyle\binom{32}{e}f^e(1-f)^{32-e}.$$</div>
</li>
</ul>
<p>We distinguish two cases, depending on whether we are baking a block to follow the shared genesis block, or subsequent blocks:</p>
<div class="math">\begin{align}
\difff{\pc, \ph, e} & := \edelay(\pc,32) - \edelaydelta(\ph, 32 - e) \tag{4}\label{eq:diff1}\\
\diff{\pc, \ph, e} & := \edelay(\pc,e) - \edelaydelta(\ph, 32 - e) \tag{5}\label{eq:diff}
\end{align}</div>
<p>Above:</p>
<ul>
<li><span class="math">\(\ph\)</span> and <span class="math">\(\pc\)</span> are parameters representing the best priorities of <span class="math">\(\hb\)</span> and <span class="math">\(\cb\)</span> respectively.</li>
<li>The equations just subtract the minimal delay function for <span class="math">\(\hb\)</span> from that of <span class="math">\(\cb\)</span>. A <em>negative value here is good for the dishonest chain <span class="math">\(\mathcal D\)</span></em>, and a <em>positive value is good for the honest chain <span class="math">\(\mathcal M\)</span></em>.</li>
<li><span class="math">\(\difff{}\)</span> corresponds to the case when we compute the delay difference for the first block <span class="math">\(\cb\)</span> bakes which is on top of genesis and <span class="math">\(\cb\)</span> has access to all endorsements for genesis, while <span class="math">\(\hb\)</span> only has its own endorsements, not those of <span class="math">\(\cb\)</span> — in contrast, <span class="math">\(\diff{}\)</span> corresponds to the case of subsequent blocks <span class="math">\(\cb\)</span> bakes; for these ones, <span class="math">\(\cb\)</span> can only use its own endorsements, since <span class="math">\(\hb\)</span><span class="quo">‘</span>s endorsements are for blocks on <span class="math">\(\mathcal M\)</span> that are not on <span class="math">\(\mathcal S\)</span>, and <span class="math">\(\cb\)</span> therefore cannot use them to extend <span class="math">\(\mathcal S\)</span>.</li>
</ul>
<p>We write sequences of priorities and endorsements of length <span class="math">\(\ell\geq 1\)</span> as
</p>
<div class="math">$$
\barpc = (\pc_1,\dots, \pc_\ell),
\qquad
\barph = (\ph_1,\dots, \ph_\ell),
\quad\text{and}\quad
\bar{e} = (e_1,\dots, e_\ell)
$$</div>
<p>
and, parameterised over these sequences, we define an <strong>accumulated difference</strong> <span class="math">\(\diffl{\barpc,\barph,\bar{e}}\)</span> by
</p>
<div class="math">$$
\diffl{\barpc, \barph, \bar{e}} := \difff{\pc_1, \ph_1, e_1} + {\sum_{2\leq i\leq \ell}}\diff{\pc_i, \ph_i, e_i}
.
$$</div>
<p><a name="forks-starting-now"></a></p>
<h2 id="forks-starting-now"><a class="toclink" href="#forks-starting-now">Forks starting now</a></h2>
<p>We can now calculate the probability that</p>
<blockquote>
<p>for chain length <span class="math">\(\ell\geq 1\)</span>, the timestamp of the head of <span class="math">\(\cb\)</span><span class="quo">‘</span>s chain, minus the timestamp of the head of <span class="math">\(\hb\)</span><span class="quo">‘</span>s chain, is equal to <span class="math">\(\delta\)</span> seconds</p>
</blockquote>
<p>as follows:</p>
<div class="math">\begin{align}
\pr_\ell[\var{chains\_diff}=\delta] := \sum_{\substack{(\barpc, \barph)\in P_2^{\ell},\bar{e}\in
[32]^{\ell}\\\diffl{\barpc,\barph,\bar{e}} =
\delta}}\;\prod_{1\leq i\leq\ell}\pendo{f,e_i}\cdot
\pprio{f,\pc_i,\ph_i} \label{eq:pr0}
\end{align}</div>
<p>
where above, <span class="math">\(P_2 = \{(p,p')\mid \text{either } p = 0 \text{ or } p'=0\}\)</span> and
</p>
<div class="math">$$
\pprio{f,\pc,\ph} :=
\left\{
\begin{array}{ll}
\pprio{f, \pc} & \text{if $\ph = 0$}\\
\pprio{1-f, \ph} & \text{if $\pc = 0$}\\
\end{array}
\right.
$$</div>
<p> to distinguish between the case when either <span class="math">\(\cb\)</span> or <span class="math">\(\hb\)</span> has priority 0.</p>
<p>Next, we give an inductive characterisation of <span class="math">\(\pr_\ell[\var{chain\_diff}=\delta]\)</span> which
is amenable to computation.</p>
<p>We first consider the probabilities corresponding to the differences in Equations \eqref{eq:diff1} and \eqref{eq:diff}.</p>
<div class="math">\begin{align}
\pr[\var{first\_diff}=\delta] := \sum_{\substack{(\pc, \ph)\in P_2,e\in
[32]\\\difff{\pc,\ph,e} =
\delta}}\;\pendo{f,e}\cdot
\pprio{f,\pc,\ph} \tag{7} \label{eq:pri00} \\
\pr[\var{subseq\_diff}=\delta] := \sum_{\substack{(\pc, \ph)\in P_2,e\in
[32]\\\diff{\pc,\ph,e} =
\delta}}\;\pendo{f,e}\cdot
\pprio{f,\pc,\ph} \tag{8} \label{eq:pri01}
\end{align}</div>
<p>Above:</p>
<ul>
<li><span class="math">\(\pr[\var{first\_diff}=\delta]\)</span> is the probability that the delay difference between the first block <span class="math">\(\cb\)</span> bakes on its secret chain <span class="math">\(\mathcal{S}\)</span> and the first block <span class="math">\(\hb\)</span> bakes on <span class="math">\(\mathcal{M}\)</span>, is <span class="math">\(\delta\)</span>.</li>
<li><span class="math">\(\pr[\var{subseq\_diff}=\delta]\)</span> is the probability that the delay difference between a block (other than the first one) <span class="math">\(\cb\)</span> bakes on its secret chain <span class="math">\(\mathcal{S}\)</span> and the block <span class="math">\(\hb\)</span> bakes on <span class="math">\(\mathcal{M}\)</span> at the same level, is <span class="math">\(\delta\)</span>.</li>
</ul>
<p>For a given difference <span class="math">\(\delta\)</span>, we define the probability of forks of length
<span class="math">\(\ell\)</span> inductively by:
</p>
<div class="math">\begin{align}
&\pr^{\mathit{now}}_1[\var{chains\_diff}=\delta] := \pr[\var{first\_diff}=\delta] \tag{9} \label{eq:prn1} \\
&\pr^{\mathit{now}}_{\ell{+}1}[\var{chains\_diff}=\delta] := \displaystyle\sum_{\substack{(\delta_{\ell}, \delta_1) \in \mathbb{Z}^2\\\delta_\ell+\delta_1=\delta}}\pr^{\mathit{now}}_{\ell}[\var{chains\_diff}=\delta_\ell]\cdot\pr[\var{subseq\_diff}=\delta_1] \tag{10} \label{eq:prnl}
\end{align}</div>
<p>
We note that <span class="math">\(\delta_\ell\)</span> can be negative as long as <span class="math">\(\delta_1\)</span> is big enough to compensate.</p>
<!--
\begin{proposition}
$\pr_\ell[\var{chains\_diff}=\delta] = \pr^{\mathit{now}}_\ell[\var{chains\_diff}=\delta].$
\end{proposition}
-->
<p>Putting this all together, the probability that <span class="math">\(\cb\)</span> bakes a fork of length <span class="math">\(\ell\)</span> that is
faster than that of <span class="math">\(\hb\)</span> is:
</p>
<div class="math">\begin{align}
\pr[\exists\mathit{faster\_fork}.(\mathit{len(faster\_fork)} = \ell)] = \displaystyle\sum_{\delta \leq 0}\pr^{\f{now}}_\ell[\var{chains\_diff}=\delta] \tag{12} \label{eq:lf} % syntax highlighting hack - !_
\end{align}</div>
<p>We use <span class="math">\(\pr[\exists\mathit{faster\_fork}.(\mathit{len(faster\_fork)}= \ell)]\)</span> to answer the question:</p>
<blockquote>
<p>Given some transaction in some block <span class="math">\(B\)</span>, how many <em>confirmations</em> — blocks after <span class="math">\(B\)</span> — must we observe to be <em>Reasonably Sure</em> that the transaction will remain in the chain?</p>
</blockquote>
<p><a name="secu"></a></p>
<p><a name="conf-nb"></a>
We call this number of blocks the <strong>confirmation number</strong>.
By <em>Reasonably Sure</em>, we mean</p>
<blockquote>
<p><span class="dquo">“</span>with probability smaller than some reasonable <strong>security threshold</strong> <span class="math">\(\secu\)</span><span class="dquo">“</span></p>
</blockquote>
<p>whose value can be fixed to the reader’s taste.
In practice, we fix a value for our security threshold <span class="math">\(\secu\)</span> such that our expectation of being wrong about a block being final is expected to be roughly once every two centuries, and conversely;
an attacker would have no reasonable confidence of seeing an attack succeed in their lifetime.
For example, when there is one block per minute, we set <span class="math">\(\secu=10^{-8}\)</span>, as done in <a href="https://research-development.nomadic-labs.com/emmy-in-the-partial-synchrony-model.html">our previous analysis of Emmy+</a>.</p>
<p>So to return to our motivating example above of the <a href="#valuable-car">Valuable Car</a>, and assuming one block per minute, all you need to do is wait <em>confirmation number</em> number of minutes before handing over the keys, and you should be Reasonably Sure that the payment of Quite A Lot is final.
If you’re inclined to be paranoid, just wait a few minutes longer (meaning in effect that you use a lower and so more stringent value of <span class="math">\(\secu\)</span>).</p>
<p>To compute the confirmation number concretely, we simply need to find the smallest
<span class="math">\(\ell\)</span> such that
</p>
<div class="math">$$
\pr[\exists\mathit{faster\_fork}.(\mathit{len(faster\_fork)} = \ell)] < \secu.
$$</div>
<p>
We built a <a href="#tool">tool for computing this, discussed below</a>.
For reference, the maths above also underlies the results presented in <a href="https://research-development.nomadic-labs.com/analysis-of-emmy.html#forks-starting-now">“forks starting now”</a>.</p>
<p><a name="eg-now"></a>
In practice — for Tezos as it currently runs Emmy+ and assuming an attacker controlling at most one fifth of the network, a message delay <a href="#delta"><span class="math">\(\Delta\)</span></a> less than 60 seconds, and a security threshold <span class="math">\(\secu\)</span> of <span class="math">\(10^{-8}\)</span> — confirmation number is about seven blocks.</p>
<p><a name="forks-starting-in-the-past"></a></p>
<h2 id="forks-started-in-the-past"><a class="toclink" href="#forks-started-in-the-past">Forks started in the past</a></h2>
<p>The analysis above can be refined using <em>conditional probabilities</em>:
as time passes we can observe the behaviour of the current blockchain <span class="math">\(\mathcal M\)</span> and gather information about its health, and this might inform our estimates of the remaining confirmation time.</p>
<p>We can ask:</p>
<blockquote>
<p>My transaction <span class="math">\(tx\)</span> was included <span class="math">\(n\)</span> blocks ago into <span class="math">\(\mathcal M\)</span> (i.e. there are <span class="math">\(n{-}1\)</span> blocks on top of the block with my transaction <span class="math">\(tx\)</span>) and <span class="math">\(\delta^o\)</span> time ago. Can I be Reasonably Sure that my <span class="math">\(tx\)</span> is final?</p>
</blockquote>
<p>To answer this, we consider the <strong>accumulated delay</strong> of the current chain <span class="math">\(\mathcal M\)</span>, relative to an <strong>ideal chain</strong> <span class="math">\(\mathcal I\)</span> operating in ideal conditions in which all blocks are produced at priority 0 (i.e. by the first available baker) with a full complement of 32 endorsements and without any network delays.</p>
<p>The accumulated delay <span class="math">\(\delta^a\)</span> of <span class="math">\(\mathcal M\)</span> is easy to calculate:
in ideal conditions the ideal chain <span class="math">\(\mathcal I\)</span> bakes one block every <span class="math">\(\mathit{time\_between\_blocks}\)</span> seconds,<sup id="fnref:tbb"><a class="footnote-ref" href="#fn:tbb">18</a></sup> so the accumulated delay of an <span class="math">\(n\)</span>-block blockchain-fragment is just</p>
<ul>
<li><span class="math">\(\delta^o\)</span> (the timestamp of the current block minus the timestamp of the block containing my transaction <span class="math">\(tx\)</span>),</li>
<li>minus <span class="math">\(n \cdot \mathit{time\_between\_blocks}\)</span>,<sup id="fnref:if-you-think-in-code"><a class="footnote-ref" href="#fn:if-you-think-in-code">19</a></sup></li>
</ul>
<p>or in symbols:
</p>
<div class="math">$$
\delta^a = \delta^o - n\cdot \mathit{time\_between\_blocks}
$$</div>
<p>The accumulated delay <span class="math">\(\delta^a\)</span> is a simple measure of the ‘health’ of <span class="math">\(\mathcal M\)</span>: the smaller <span class="math">\(\delta^a\)</span> is, the healthier <span class="math">\(\mathcal M\)</span> is.
Intuitively an unhealthy chain is easier for a hostile baker <span class="math">\(\cb\)</span> to attack with a hostile chain <span class="math">\(\mathcal S\)</span>, so our task is now to quantify this.</p>
<p>We compute the probability that our hostile baker <span class="math">\(\cb\)</span> has a hostile chain <span class="math">\(\mathcal S\)</span> that <span class="math">\(\cb\)</span> has forked from the main chain just before the block with our transaction <span class="math">\(tx\)</span> and that has the same length as <span class="math">\(\mathcal M\)</span> but is faster.
We conceptually split <span class="math">\(\mathcal S\)</span> into two parts: a “past” one consisting of first <span class="math">\(n\)</span> blocks, and a “future” one consisting of the subsequent blocks.</p>
<p>We will use our ideal chain <span class="math">\(\mathcal I\)</span> as an intermediate reference step.
Recall that we assume that blocks on <span class="math">\(\mathcal I\)</span> are baked with no delay, that is, every <span class="math">\(\bd\)</span> seconds.
Then we proceed in three steps:</p>
<ol>
<li>For the first <span class="math">\(n\)</span> blocks, we compare <span class="math">\(\cb\)</span><span class="quo">‘</span>s chain <span class="math">\(\mathcal S\)</span> with the ideal chain
<span class="math">\(\mathcal I\)</span>.</li>
<li>We then shift from <span class="math">\(\mathcal I\)</span> to <span class="math">\(\mathcal M\)</span> to account for <span class="math">\(\delta^a\)</span>.</li>
<li>At this point, we are in a similar situation as with <a href="#forks-starting-now">forks starting
now</a>, and so we compare <span class="math">\(\mathcal S\)</span> with <span class="math">\(\mathcal M\)</span>.</li>
</ol>
<p>For Item 1, the basic element we need is the difference between the minimal
block delays on <span class="math">\(\mathcal S\)</span> and on <span class="math">\(\mathcal I\)</span>:</p>
<div class="math">\begin{align}
\difffp{\pc, \ph, e} & := \edelay(\pc,32) - \mathit{time\_between\_blocks} \tag{13} \label{eq:pdiff1}\\
\diffp{\pc, \ph, e} & := \edelay(\pc,e) - \mathit{time\_between\_blocks} \tag{14} \label{eq:pdiff}
\end{align}</div>
<p>We note that the above equations are similar to
Equations \eqref{eq:diff1} and \eqref{eq:diff}, with the difference that we
subtract <span class="math">\(\bd\)</span> from the definition of the ideal chain.</p>
<p>We can write the probabilities corresponding to Equations \eqref{eq:pdiff1} and \eqref{eq:pdiff} as follows:</p>
<div class="math">\begin{align}
&\pr^{\pcst}[\f{first\_diff}=\delta] = \displaystyle\sum^{32}_{e=0}\pendo{f,e}\cdot
\sum_{\substack{\pc\geq 0\\{\difffp{\pc,0,e}=\delta}}}
\pprio{f,\pc} \tag{15} \label{eq:pasteq1} \\
&\pr^{\pcst}[\f{subseq\_diff}=\delta] = \displaystyle\sum^{32}_{e=0}\pendo{f,e}\cdot
\sum_{\substack{\pc\geq 0\\{\diffp{\pc,0,e}=\delta}}}
\pprio{f,\pc} \tag{16} \label{eq:pastge1}
\end{align}</div>
<p>Similar to Equations \eqref{eq:prn1} and \eqref{eq:prnl}, for a given difference <span class="math">\(\delta\)</span>,
the probability that the difference between <span class="math">\(\mathcal S\)</span> and <span class="math">\(\mathcal I\)</span> is <span class="math">\(\delta\)</span> is defined inductively as:</p>
<div class="math">\begin{align}
&\pr^{\pcst}_1[\f{chains\_diff} = \delta] = \pr^{\pcst}[\f{first\_diff} = \delta] \tag{17} \label{eq:past1} \\
&\pr^{\pcst}_n[\f{chains\_diff} = \delta] =
\displaystyle\sum_{\substack{(\delta_{n{-}1}, \delta_1) \in
\mathbb{Z}^2\\\delta_{n{-}1}+\delta_1=\delta}}\pr^\pcst_{n{-}1}[\f{chains\_diff} = \delta_{n{-}1}]\cdot\pr^\pcst[\f{subseq\_diff} = \delta_1]
\end{align}</div>
<p>For Item 2, <span class="math">\(\pr_n^\pcst[\f{chains\_diff} = \delta - \delta^a]\)</span> captures the
shift with respect to <span class="math">\(\delta^a\)</span> and this
probability is the base case in the inductive definition for the probability
that the difference between <span class="math">\(\mathcal S\)</span> and <span class="math">\(\mathcal M\)</span> is <span class="math">\(\delta\)</span>, which is
the probability mentioned in Item 3:</p>
<div class="math">\begin{align}
&\pr^\pcstn_1[\f{chains\_diff} = \delta \mid \delta^a,n] = \pr_n^\pcst[\f{chains\_diff} = \delta - \delta^a]\tag{18}\label{eq:shift}\\
&\pr^\pcstn_{\ell{+}1}[\f{chains\_diff} = \delta \mid \delta^a,n] =
\displaystyle\sum_{\substack{(\delta_{\ell}, \delta_1) \in
\mathbb{Z}^2\\\delta_\ell+\delta_1=\delta\\\delta_{\ell} > 0}}\pr^\pcstn_{\ell}[\f{chains\_diff} = \delta_\ell\,\mid \delta^a,n]\cdot\pr_1^\pcstn[\f{chains\_diff} = \delta_1] \tag{19} \label{eq:ppl}
\end{align}</div>
<p>For brevity we will write</p>
<ul>
<li><span class="math">\(\pr^\pcst\)</span> for the probability that <span class="math">\(\cb\)</span> baked <span class="math">\(n\)</span> blocks until now and</li>
<li><span class="math">\(\pr^\pcstn\)</span> for the conditional probability that <span class="math">\(\cb\)</span> bakes <span class="math">\(l\)</span> blocks from now given <span class="math">\(\pr^\pcst\)</span> and the accumulated delay.</li>
</ul>
<p>Note the condition <span class="math">\(\delta_{\ell} > 0\)</span> in Equation \eqref{eq:ppl}:
since <span class="math">\(\delta\)</span> represents the difference between the timestamps of <span class="math">\(\hb\)</span> and <span class="math">\(\cb\)</span>, by means of this condition, we do not take into account the probabilities of forks of length <span class="math">\(\ell\)</span> the attacker could have won.
Thus the probability in Equation \eqref{eq:ppl} represents the probability that the difference between the timestamps of <span class="math">\(\mathcal S\)</span> and <span class="math">\(\mathcal M\)</span> is <span class="math">\(\delta\)</span> and any prefix of <span class="math">\(\mathcal S\)</span> is not faster than the corresponding prefix of <span class="math">\(\mathcal M\)</span>. In other words, <span class="math">\(\ell{+}1\)</span> is the first level at which the attacker might have a faster chain.</p>
<p>We define <span class="math">\(\pr[\exists\mathit{faster\_fork}.(\mathit{len(faster\_fork)}\leq \ell) \mid \delta^a,n]\)</span>
to be the probability the attacker has a faster fork of <em>any</em> length smaller than or equal
to <span class="math">\(\ell\)</span>:</p>
<div class="math">\begin{align}
\pr[\exists\mathit{faster\_fork}.(\mathit{len(faster\_fork)}\leq \ell) \mid \delta^a,n] = \displaystyle\sum_i^{\ell}\sum_{\delta \leq 0}\pr^\f{past}_i[\f{chains\_diff} = \delta \mid \delta^a,n].
\end{align}</div>
<p>For a block to be considered final, given an accumulated delay <span class="math">\(\delta^a\)</span> we
need to find the smallest <span class="math">\(n\)</span> such that the probability to have a fork of <em>any</em>
length in the future is smaller than our <a href="#secu">security threshold</a> <span class="math">\(\secu\)</span> (e.g. <span class="math">\(\secu=10^{-8}\)</span>), namely, we need to find the smallest <span class="math">\(n\)</span> such that <span class="math">\(\forall \ell. \pr[\exists\mathit{faster\_fork}.(\mathit{len(faster\_fork)}\leq \ell) \mid \delta^a,n] < \secu\)</span>.
To effectively capture the universal quantifier, we define <span class="math">\(\pr[\exists\mathit{faster\_fork} \mid \delta^a,n]\)</span> as the following limit:</p>
<div class="math">\begin{align}
\tag{20}
\label{eq:pfl}
\pr[\exists \mathit{faster\_fork} \mid \delta^a,n] =
\displaystyle{\lim_{\ell \rightarrow \infty}} \pr[\exists\mathit{faster\_fork}.(\mathit{len(faster\_fork)}\leq \ell) \mid \delta^a,n].
\end{align}</div>
<p>To solve Equation \eqref{eq:pfl}, it suffices to compute the least <span class="math">\(\ell\)</span> such that:
</p>
<div class="math">\begin{align}
\frac{\pr[\exists\mathit{faster\_fork}.(\mathit{len(faster\_fork)}\leq \ell) \mid
\delta^a,n]}{\pr[\exists\mathit{faster\_fork}.(\mathit{len(faster\_fork)}\leq \ell{-}1) \mid \delta^a,n]} \leq 1 +
\epsilon. \tag{21} \label{eq:pastf}
\end{align}</div>
<p>
where we introduce <span class="math">\(\epsilon\)</span> to denote some desired computational precision, just to limit the computation.
That is, the larger <span class="math">\(\ell\)</span> is, the smaller the probability, and
at a certain point the probability of a fork of length <span class="math">\(\ell\)</span> is within <span class="math">\(\epsilon\)</span> of the probability of a fork of length <span class="math">\(\ell{-}1\)</span>, and we decide that we are now precise enough and can stop the computation.</p>
<p>So now, to compute (or rather: to estimate using precision <span class="math">\(\epsilon\)</span>) the <a href="#conf-nb">confirmation number</a> for a given
accumulated delay <span class="math">\(\delta^a\)</span>, we just need to find the smallest <span class="math">\(n\)</span> such that
</p>
<div class="math">$$
\pr[\exists \mathit{faster\_fork} \mid \delta^a,n] < \secu.
$$</div>
<p>
Calculating this is best done by computer, so we have built …</p>
<h2 id="a-tool-and-the-attack-model"><a class="toclink" href="#a-tool-and-the-attack-model">A tool, and the attack model</a></h2>
<p><a name="tool"></a></p>
<h3 id="a-concrete-tool"><a class="toclink" href="#a-concrete-tool">A concrete tool</a></h3>
<p>The methodology above underlies the results presented in <a href="https://research-development.nomadic-labs.com/analysis-of-emmy.html#forks-started-in-the-past">a previous blogpost on
forks started in the
past</a>.
Since then, we have created a <a href="https://gitlab.com/nomadic-labs/emmyplus-experiments/-/tree/emmystar_exp">standalone
tool</a>
to perform relevant calculations, which we have made <a href="https://nomadic-labs.gitlab.io/emmyplus-experiments/">accessible online as a web
demo</a> for forks started in
the past.</p>
<p>We can use our tool to compute and then compare concrete confirmation numbers for both the <a href="#forks-starting-now">forks starting now</a> and the <a href="#forks-starting-in-the-past">forks started in the past</a> scenarios considered above.
Note that forks starting in the past is simply a conditional probability: <em>forks starting now</em> makes no assumptions about future chain health, whereas <em>forks starting in the past</em> asks “<em>given</em> a particular value for chain health since the transaction concerned, what is the expected confirmation number”?
So recall our <a href="#eg-now">previous assumptions</a> of</p>
<ul>
<li>an attacker controlling at most one fifth of the network and</li>
<li>a security threshold <span class="math">\(\secu\)</span> of <span class="math">\(10^{-8}\)</span>.</li>
</ul>
<p>Under a <em>forks starting now</em> scenario — that is, at the time that we generated our block — we expected to wait <a href="#eg-now">seven blocks</a> before considering our transaction final.</p>
<p>Suppose now that time has passed and we see that since our transaction, three blocks have been baked in 190 seconds — i.e. with a 10 second delay with respect to an ideal chain in Emmy+ which bakes one block every 60 seconds.
Then our tool tells us that, assuming <a href="#delta"><span class="math">\(\Delta\)</span></a> is less than 60 seconds, the confirmation number is in fact three, so we can already consider our transaction final — four blocks earlier than our original worst-case scenario.
And this is just because we have observed that the chain has been reasonably healthy since it included our transaction.</p>
<p>Thus by taking account of chain health since our transaction was included, we may — if the chain remains healthy — get a smaller number of confirmations than in the case of forks starting now.</p>
<h3 id="notes-on-the-attack-model"><a class="toclink" href="#notes-on-the-attack-model">Notes on the attack model</a></h3>
<p>As always, our security guarantees depend on our attack model:
i.e. what we assume an attacker wants to accomplish, and what powers we assume the attacker has when trying to do so.
So it is as well to be clear about the limitations, as follows:</p>
<ol>
<li>In the case of <a href="#forks-starting-now">forks starting now</a>, we have only considered that the attacker’s goal is to undo a transaction, so the attack stops when the attacker succeeds.
In an alternative attack model, the attacker could keep switching branches and start again with the purpose of maintaining a fork for as long as possible.
(Don’t worry!
We have analysed this scenario in <a href="https://research-development.nomadic-labs.com/the-case-of-mixed-forks-in-emmy.html">our blog post on mixed forks in Emmy</a>.)
The attacker could also play on multiple branches, while we consider only two branches.</li>
<li>In the case of <a href="#forks-starting-in-the-past">forks started in the past</a>, we have only considered the case when the attacker starts a fork <em>at the block with the transaction to be undone</em>.
In an alternative attack model, the attacker could have started a fork before the block with the transaction to be undone — though one can always bring this within the scope of our analysis by assuming the attacker is trying to undo an arbitrary transaction in the earlier block.</li>
</ol>
<p><a name="future"></a></p>
<h2 id="the-future"><a class="toclink" href="#the-future">The future</a></h2>
<p>Tezos plans to switch to the classical <span class="caps">BFT</span>-style consensus algorithm Tenderbake, because <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">this offers faster finality</a>.
Does that mean the Emmy family of algorithms is obsolete?
Yes if you only care about Tezos, but <em>absolutely not</em>, if you care about Nakamoto-style consensus in general.</p>
<p>Emmy* is a perfectly fine consensus algorithm which arguably represents an evolutionary peak in Nakamoto-style consensus: the fact that it and Tezos stand to part ways reflects more on what Tezos requires for its future, than on Emmy itself.</p>
<p>Furthermore, all Nakamoto-style consensus algorithms are quite similar, so the lessons learned from optimising Nakamoto-style consensus to run the Tezos blockchain for seven years with increasing sophistication and real-world reliability may be valid, or at least indicative, of <em>your</em> favourite Nakamoto-style consensus algorithm too.
Mathematics doesn’t rust, and if the maths in this blog post might help inform the design of future blockchain protocols, then it will have done its job.</p>
<h3 id="further-reading"><a class="toclink" href="#further-reading">Further reading:</a></h3>
<ul>
<li>More examples are in our previous blog posts on <a href="https://research-development.nomadic-labs.com/emmy-in-the-partial-synchrony-model.html">Emmy+</a> and <a href="https://research-development.nomadic-labs.com/faster-finality-with-emmy.html">Emmy*</a>.</li>
<li>Practical aspects of the implementation are discussed <a href="https://research-development.nomadic-labs.com/on-defending-against-malicious-reorgs-in-tezos-proof-of-stake.html#on-the-methodology">here</a>.</li>
<li>A generalisation of forks starting now is discussed in <a href="https://research-development.nomadic-labs.com/the-case-of-mixed-forks-in-emmy.html">a blog post on the case of mixed forks</a>.</li>
<li><a href="https://tezos.gitlab.io/009/proof_of_stake.html">More Emmy-related documentation is here</a>.</li>
</ul>
<p><strong>Acknowledgements</strong>: We thank <a href="https://prosecco.gforge.inria.fr/personal/bblanche/index-eng.html">Bruno Blanchet (<span class="caps">INRIA</span>)</a> for providing the framework for the analyses in this post.
Arthur Breitman made most of the design choices in the Emmy family described above.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:one-true">
<p>Actually, it’s The One True Blockchain, like strawberry jam is The One True Jam [<em>not quince</em> -ed] and Breaking Bad and Better Call Saul are The Joint One True <span class="caps">TV</span> Series. But, we digress. <a class="footnote-backref" href="#fnref:one-true" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:no-hyperbole">
<p>This is not hyperbole. It was. <a class="footnote-backref" href="#fnref:no-hyperbole" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:subtle">
<p>Like, really subtle. Please don’t expect to read this blog post in a hurry: if you’re not familiar with the material then it <em>will</em> force you to think. <a class="footnote-backref" href="#fnref:subtle" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:name">
<p>The consensus algorithm was unnamed in the whitepaper referenced. The name ‘Emmy’ was chosen later, in the <a href="https://research-development.nomadic-labs.com/emmy-an-improved-consensus-algorithm.html">blog post introducing Emmy+</a>. <a class="footnote-backref" href="#fnref:name" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:emmystarsecure">
<p>The upgrade from Emmy to Emmy+ improved security and did not degrade performance. Similarly for the forthcoming upgrade to <a href="https://gitlab.com/tezos/tzip/-/blob/master/drafts/current/draft_emmy-star.md">Emmy*</a>. <a class="footnote-backref" href="#fnref:emmystarsecure" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:time">
<p>It is a whole other issue to determine whether timestamp clocks are in sync, which is outside the scope of this blog post. We will just note that the system is engineered to permit a discrepancy of up to a certain number of seconds, beyond which two timestamps are deemed not equal in some strong sense (15 seconds in Emmy+; 5 seconds in Emmy*). <a class="footnote-backref" href="#fnref:time" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:how-allocated">
<p>Enumerate active participants as <span class="math">\((a_1,a_2,\dots,a_n)\)</span> and to every nonnegative number <span class="math">\(j\geq 0\)</span> assign a participant <span class="math">\(f(j)\in\{a_1,a_2,\dots,a_n\}\)</span> with probability proportional to participants’ stake — meaning that if <span class="math">\(a_1\)</span> has twice the stake of <span class="math">\(a_2\)</span>, then the probability that <span class="math">\(f(j)=a_1\)</span> is twice the probability that <span class="math">\(f(j)=a_2\)</span>.
Then the earliest priority <span class="math">\(p(a_i)\)</span> is the <em>least</em> <span class="math">\(j\)</span> such that <span class="math">\(f(j)=a_i\)</span>.
<br>
In practice, all we care about is the first two participants — that is, the <span class="math">\(a_i\)</span> such that <span class="math">\(p(a_i)=0\)</span> (the head position in the queue) and the <span class="math">\(a_j\)</span> such that <span class="math">\(p(a_j)\)</span> is least nonzero (the next position in the queue), since these are the priorities which are most relevant in practice. <a class="footnote-backref" href="#fnref:how-allocated" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn:distributed">
<p>This is a distributed network, so different participants may see different sets of endorsements; but let’s look at this from the point of view of a particular participant. <a class="footnote-backref" href="#fnref:distributed" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
<li id="fn:academia">
<p>Any resemblance to multiple self-citing research cliques in academia, competing to do the same research but with different terminology and while assiduously ignoring each other, is coincidental, of course. But in truth, the selfish clique is a universal problem, as is how to design systems that efficiently and robustly reward cooperation above whatever clique or echo-chamber can most stridently self-cite. <a class="footnote-backref" href="#fnref:academia" title="Jump back to footnote 9 in the text">↩</a></p>
</li>
<li id="fn:tenderbake-no-fork">
<p>The planned upgrade to <a href="https://arxiv.org/abs/2001.11965">Tenderbake</a> would make forks impossible. See <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">the blog post</a>. <a class="footnote-backref" href="#fnref:tenderbake-no-fork" title="Jump back to footnote 10 in the text">↩</a></p>
</li>
<li id="fn:pos">
<p>The difference is that Bitcoin is proof-of-work (PoW): it is computationally <em>extremely expensive</em> to generate chains, so (in theory) hostile forks won’t happen because beyond a certain number of blocks they would simply cost too much to create. Emmy is proof-of-stake (PoS): it is computationally cheap to generate chains, but participants with a large stake in the blockchain get more baking slots, and they have no incentive to undermine their own (larger) stake.
<br>
One might say that attacking a PoW system is <em>hard</em>, whereas attacking a PoS system is <em>silly</em>.<sup id="fnref:stupid"><a class="footnote-ref" href="#fn:stupid">12</a></sup> <a class="footnote-backref" href="#fnref:pos" title="Jump back to footnote 11 in the text">↩</a></p>
</li>
<li id="fn:stupid">
<p>Though as always, it depends on the attack model. In principle, a large actor with effectively unlimited resources and not motivated by financial gain could acquire a large stake in an unpermissioned (i.e. open to participation by all) proof-of-stake or a proof-of-work system <em>in order</em> to break it. <a class="footnote-backref" href="#fnref:stupid" title="Jump back to footnote 12 in the text">↩</a></p>
</li>
<li id="fn:economist">
<p>An economist might ask: “What is a suitable <a href="https://en.wikipedia.org/wiki/Utility"><em>utility function</em></a>?”. <a class="footnote-backref" href="#fnref:economist" title="Jump back to footnote 13 in the text">↩</a></p>
</li>
<li id="fn:same-length">
<p>To be precise, participants bake on a given chain until they observe a strictly longer one. In particular, if more than one chain exists with the same length, then participants will <em>not</em> switch — but then they may all converge to bake on whichever of the chains subsequently bakes a block and so becomes longest. <a class="footnote-backref" href="#fnref:same-length" title="Jump back to footnote 14 in the text">↩</a></p>
</li>
<li id="fn:statistics">
<p>At time of writing, the Tezos live chain has about 80,000 rolls (where 1 roll = 8,000 ꜩ).
Statistics are on <a href="https://tzstats.com">TZStats</a>; divide “Active Staking” by 8,000.
Endorsement slots are distributed to participants in proportion to their stake, so (simplifying slightly): each roll has an equal chance of getting an endorsement slot at each level, and assuming all is running smoothly, the Tezos blockchain produces one level per minute under Emmy and Emmy+ and two levels per minute under Emmy*.
So working with these numbers, 32 of the 80,000 rolls get an endorsement slot every minute; that’s 0.04%. Emmy* will increase this to 256 out of 80,000 per level; making 0.32%.
Very roughly speaking, this means that if you hold 1 roll = 8,000 ꜩ then</p>
<ul>
<li>under Emmy and Emmy+ you have roughly a one in two chance each day of being invited to endorse a block (<span class="math">\(1 - (1-0.0004)^{1440}\approx 0.44\)</span>), because there are <span class="math">\(1440 = 24 * 60\)</span> levels (i.e. blocks produced) per day.</li>
<li>Under Emmy*, this chance rises to virtual certainty (<span class="math">\(1 - (1-0.0032)^{2880}\approx 0.99\)</span>).</li>
</ul>
<p>Aside from more user participation, this is also expected to reduce <a href="https://research-development.nomadic-labs.com/emmy-in-the-partial-synchrony-model.html">confirmation times</a>, making the blockchain as a whole more stable. <a class="footnote-backref" href="#fnref:statistics" title="Jump back to footnote 15 in the text">↩</a></p>
</li>
<li id="fn:ell">
<p>In the more general case, we would just offset <span class="math">\(\ell\)</span> by some amount to reflect the length of any existing blockchain. It makes no difference to the analysis. <a class="footnote-backref" href="#fnref:ell" title="Jump back to footnote 16 in the text">↩</a></p>
</li>
<li id="fn:calculate">
<p>This equation can be read as follows:</p>
<ul>
<li>Once a block is baked, it takes <span class="math">\(\Delta\)</span> time units for the other honest bakers to receive it,</li>
<li>and another <span class="math">\(\Delta\)</span> time units for their endorsements of that block to get back to <span class="math">\(\hb\)</span>.</li>
<li>Therefore, the earliest baking time for the next block is <span class="math">\(\max(2\Delta, \edelay(\ph, e))\)</span>.</li>
<li>If <span class="math">\(\hb\)</span> prefers to bake its block without endorsements, then the earliest baking time is <span class="math">\(\max(\Delta, \edelay(\ph, 0))\)</span>.</li>
</ul>
<p><a class="footnote-backref" href="#fnref:calculate" title="Jump back to footnote 17 in the text">↩</a></p>
</li>
<li id="fn:tbb">
<p>In Emmy and Emmy+, <span class="math">\(\mathit{time\_between\_blocks}\)</span> is one minute (<span class="math">\(\bd\)</span>). In Emmy*, <span class="math">\(\mathit{time\_between\_blocks}\)</span> is 30 seconds (<span class="math">\(\md\)</span>). <a class="footnote-backref" href="#fnref:tbb" title="Jump back to footnote 18 in the text">↩</a></p>
</li>
<li id="fn:if-you-think-in-code">
<p>That’s <code>(ts(current_blk) - ts(blk_with_tx)) - n*tbb</code>, for those who think in code, setting <code>tbb</code> to either <code>bd</code> or <code>md</code> depending on whether the analysis is for Emmy, Emmy+, or Emmy*. <a class="footnote-backref" href="#fnref:if-you-think-in-code" title="Jump back to footnote 19 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Three questions to Nomadic Labs interns — Mathis Gontier Delaunay2021-07-28T14:00:00+02:002021-07-28T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-07-28:/three-questions-to-nomadic-labs-interns-mathis-gontier-delaunay.html<p>Short interview with Mathis Gontier Delaunay</p><p>Each year we host around half-a-dozen interns (<em>stagiaires</em>), for periods of two to six months.
These young men and women are usually studying or have just finished studying for undergraduate or Master’s degrees, and are eager to gain experience in the blockchain industry.
Each has a unique story to tell about their individual interest in this technology.</p>
<p>In this blogpost, we will ask three questions of one of our current interns: <strong>Mathis Gontier Delaunay</strong> (and a couple of questions of his mentors at Nomadic Labs).</p>
<p>Mathis — so happy to have you with us!
We hope you’re having a wonderful and educational period with Nomadic Labs.
We’d love to hear a bit about you and your activities <em>chez nous</em> …</p>
<h1 id="questions-for-mathis"><a class="toclink" href="#questions-for-mathis">Questions for Mathis</a></h1>
<h2 id="1-please-present-yourself-and-your-academic-background"><a class="toclink" href="#1-please-present-yourself-and-your-academic-background">1. Please present yourself and your academic background</a></h2>
<p>My name is Mathis Gontier Delaunay and I am 21.</p>
<p>I took a Mathematics and Physics preparatory class in Rennes with the Computer Science option, which included algorithms, programming methods, and data structures with OCaml.
This provided the grounding for my current interest in computer science. </p>
<p>Since September 2020 I have been studying at <a href="https://www.telecom-sudparis.eu/en/">Telecom SudParis</a>, a generalist engineering school oriented in the digital sector. </p>
<p>In September 2020 I also joined <a href="https://kryptosphere.org/en/"><span class="caps">KRYPTOSPHERE</span></a>, the first French student association dedicated to blockchain technologies and cryptocurrencies.
I became passionate about the technology, its associated ideology, and its fast-growing ecosystem.
I find the underlying concepts fascinating, and I am more and more drawn to the engineering aspects of its applications and protocols.
I really believe in the future of these technologies, which can solve many problems, and I want to understand them in depth to participate in their development.</p>
<p>In April 2021 I was delighted to begin a developer’s role with a team of three researchers and one student working on a decentralized finance protocol that aims to create a “zero spread money market”; a much more efficient way to use liquidity than current lending protocols.
The project has since grown and seven people are working on it today, with a proof of concept on the way: <a href="https://morpho.best/">https://morpho.best/</a>.</p>
<p>I hope one day to fully understand decentralized ledger technology and to participate in its long-term development.
I look forward to contributing to the maturation of decentralized ledgers into a practical technology and a useful part of our lives and society. </p>
<h2 id="2-tell-us-more-about-your-internship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months"><a class="toclink" href="#2-tell-us-more-about-your-internship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months">2. Tell us more about your internship: main subject, who is your mentor, what you have learned and especially, what surprised you the most within these months?</a></h2>
<p>In early June 2021 I joined the Umami team at Nomadic Labs, and I will stay until early September.
<a href="https://umamiwallet.com/">Umami</a> is a desktop cryptocurrency wallet for the Tezos ecosystem whose Beta version was launched in April.
It is developed by a team of about ten people at Nomadic Labs, currently working on new features like hardware wallet integration and the interaction with Tezos decentralized apps (<em>Dapps</em>) <a href="https://www.tezosprojects.com/">as listed on this webpage</a>. </p>
<p>For my internship I’m contributing to Umami’s development.
My mentor is Rémy El Sibaïe, one of the five developers of the team. </p>
<p>I bring my understanding of blockchain technology operation, usage and applications to my role, and this internship is an opportunity for me to participate with a team in actual software development in a blockchain ecosystem. </p>
<p>A big part of this turns out to be learning specialist tooling and collaboration techniques! For example, I spent my first days getting used to <a href="https://git-scm.com/book/en/v2/Getting-Started-About-Version-Control">Git version control</a>, and learning how to work with a team of developers on the same project.
That’s two valuable life skills learned right there in the first week!</p>
<p>Umami is written in <a href="https://reasonml.github.io/en/">ReasonML</a>, a functional language which works much like OCaml (and notably having a strong type system) but with a Javascript-like syntax, and which integrates better with the <a href="https://reactjs.org/">React library</a>.
I now have a better understanding of functional languages and how to use them to build applications like Umami.
My mentor also explained to me the more precise techniques that are used in Umami, such as <a href="https://en.wikipedia.org/wiki/Futures_and_promises">futures and promises</a>.</p>
<p>Contributing to an open-source project is very rewarding: after only one month, my first contribution to the project is already in the main codebase and can be used by the community.
But working here at Nomadic Labs, surrounded and supervised by blockchain experts, with a really strong understanding of the technology and its challenge, would already have been an accomplishment for this first internship.
Listening to their discussions about the future of Tezos is a pleasure each day. </p>
<h2 id="3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-internship"><a class="toclink" href="#3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-internship">3. Why did you choose to become a part of the blockchain ecosystem and Nomadic labs? What are your plans after completing this internship?</a></h2>
<p>I have a strong interest in computer science and particularly in decentralized ledger technologies, which for me have opened the door to many innovations that I expect to revolutionize the way we exchange value, finance, and a lot more.
I naturally wanted to do my first year internship in this domain.</p>
<p>Nomadic Labs is a major player in the crypto-blockchain sector in France, and also one of the most technical ones. Nomadic Labs is a key contributor to the core protocol of the really innovative blockchain that is Tezos, and combines this with wider work on adoption and support.
That combination is what attracted me. </p>
<p>Participating in the development of an ecosystem with infinite potential is very exciting and motivating, and this experience has confirmed my attraction to the sector. That is why I would like to specialize in distributed systems as soon as possible, probably at university, in France or abroad.
I would also like to conduct a bigger research / development project for my next internship.
Who knows: perhaps that too could be at Nomadic Labs. </p>
<h1 id="questions-for-mathis-nomadic-labs-mentor-remy-el-sibai"><a class="toclink" href="#questions-for-mathis-nomadic-labs-mentor-remy-el-sibai">Questions for Mathis’ Nomadic Labs mentor Rémy El Sibaï:</a></h1>
<h2 id="what-is-your-input-to-the-work-of-mathis"><a class="toclink" href="#what-is-your-input-to-the-work-of-mathis">What is your input to the work of Mathis?</a></h2>
<p>Mathis has a good understanding of the blockchain world and has been making useful contributions to the Umami project from the start of his internship. </p>
<p>This is his first industrial experience, so his learning curve has included our development tools and professional good practice. And of course, Mathis has been learning the functional programming and type systems that are specific to OCaml and ReasonML. </p>
<p>Mathis has good ideas and a sound understanding of users’ requirements. We look forward to integrating his contribution into our production code.</p>
<h2 id="why-is-the-topic-of-this-internship-important"><a class="toclink" href="#why-is-the-topic-of-this-internship-important">Why is the topic of this internship important?</a></h2>
<p>Mathis is making substantive contributions to core features of the Umami system and really pulling his weight (= making significant contributions) as he works with our team. We could not ask more of an intern.</p>
<p>Mathis also brings something else of great value to us: he sees our project with fresh eyes, and this helps us to find and fix all the accessibility mistakes which we — as the system’s developers who know Umami too well — are liable to miss.
Thank you Mathis! </p>A tale of two reductions in gas consumption in Tezos2021-07-26T15:00:00+02:002021-07-26T15:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-07-26:/a-tale-of-two-reductions-in-gas-consumption-in-tezos.html<p>Two stories about reducing gas consumption in Tezos</p><ul>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#tale-1-a-new-tail-recursive-monadic-interpreter">Tale 1: a new tail recursive monadic interpreter</a><ul>
<li><a href="#an-aside-side-effects-via-monads">An aside: side-effects via monads</a></li>
<li><a href="#an-interpreter-for-michelson-scripts">An interpreter for Michelson scripts</a><ul>
<li><a href="#michelbaby-typechecking-for-free-courtesy-of-the-ocaml-typechecker">MichelBaby typechecking for free, courtesy of the OCaml typechecker</a></li>
<li><a href="#the-step-function-version-1">The <code>step</code> function, Version 1</a></li>
<li><a href="#the-step-function-version-11">The <code>step</code> function, Version 1.1</a></li>
</ul>
</li>
<li><a href="#making-the-interpreter-fast">Making the interpreter fast</a><ul>
<li><a href="#reducing-the-number-of-binds">Reducing the number of <code>bind</code>s</a></li>
<li><a href="#the-step-function-version-2">The <code>step</code> function, Version 2</a></li>
</ul>
</li>
<li><a href="#is-that-everything">Is that everything?</a></li>
</ul>
</li>
<li><a href="#tale-2-hunt-for-a-serialization-performance-bug">Tale 2: hunt for a serialization performance bug</a><ul>
<li><a href="#doing-serialization-right-thanks-to-well-typed-encodings">Doing serialization right thanks to well-typed encodings</a></li>
<li><a href="#how-costly-is-this-abstraction">How costly is this abstraction?</a></li>
<li><a href="#how-to-fix-fix">How to fix <code>fix</code>?</a></li>
<li><a href="#is-that-everything_1">Is that everything?</a></li>
</ul>
</li>
<li><a href="#conclusion">Conclusion</a></li>
</ul>
<h2 id="introduction"><a class="toclink" href="#introduction">Introduction</a></h2>
<p>We subscribe to <a href="https://en.wikipedia.org/wiki/Kent_Beck">Kent Beck’s</a> motto:</p>
<blockquote>
<p><em>Make it work, make it right, make it fast!</em></p>
</blockquote>
<p>— in that order.
This implies an <em>incremental</em> software development process:</p>
<ul>
<li>When you <em>make it work</em>, you may elide corner cases or design considerations required to <em>make it right</em>; and</li>
<li>when you <em>make it right</em>, you may introduce abstractions which are good for provability but are too naive to <em>make it fast</em> in the implementation.</li>
</ul>
<p>This is normal: incremental development separates concerns so we can properly deal with each of them.</p>
<p>We’d like to share with you two tales of making Tezos faster in Granada by following the motto above:<sup id="fnref:economic-protocol"><a class="footnote-ref" href="#fn:economic-protocol">1</a></sup></p>
<ul>
<li><a href="#tale1">Tale 1</a> led to an optimization of the Michelson interpreter; and</li>
<li><a href="#tale2">Tale 2</a> led to an optimization of the serialization layer.</li>
</ul>
<p>Both tales are set after the “make it right” stage and during the “make it fast” stage.
The code was clean (and we wanted to keep it that way);
we just wanted to increase performance.</p>
<p>With these two optimizations,
<a href="https://tezos.gitlab.io/protocols/010_granada.html">Granada</a> — the <a href="https://tzstats.com/election/29#promotion_period">latest, and recently-accepted, amendment proposal of Tezos economic protocol</a> —
executes smart contracts with much better gas consumption.<sup id="fnref:gas"><a class="footnote-ref" href="#fn:gas">2</a></sup>
For example, a typical call to Dexter entrypoint <code>XtzToToken</code> consumes roughly</p>
<ul>
<li>75600 units of gas in Florence, and</li>
<li>9320 units of gas in Granada.</li>
</ul>
<p>That’s an eightfold improvement.<sup id="fnref:test-1"><a class="footnote-ref" href="#fn:test-1">3</a></sup></p>
<p><a name="tale1"></a></p>
<h2 id="tale-1-a-new-tail-recursive-monadic-interpreter"><a class="toclink" href="#tale-1-a-new-tail-recursive-monadic-interpreter">Tale 1: a new tail recursive monadic interpreter</a></h2>
<p>Tezos executes smart contracts in <a href="https://tezos.gitlab.io/active/michelson.html">Michelson</a>,
a strongly-typed low-level language.<sup id="fnref:bytecode"><a class="footnote-ref" href="#fn:bytecode">4</a></sup>
Roughly speaking, when provided with </p>
<ul>
<li>an <em>input argument</em>, and </li>
<li>a <em>storage state</em>, and</li>
<li>a stack of <em>pending operations</em>,</li>
</ul>
<p>a Michelson script computes</p>
<ul>
<li>a new storage state, and</li>
<li>a new stack of pending operations, and</li>
<li>it may also carry out some concrete action to change the state of the blockchain — the technical term is <a href="https://en.wikipedia.org/wiki/Side_effect_(computer_science)">a side-effect</a>. </li>
</ul>
<h3 id="an-aside-side-effects-via-monads"><a class="toclink" href="#an-aside-side-effects-via-monads">An aside: side-effects via monads</a></h3>
<p>Monads are how functional programmers deal with effectful computation, meaning programs that may compute a return value and also <em>have</em> side-effects on the ambient world as they do so (input/output, token transfers, etc).
The impatient reader is welcome to <a href="#interp">skip forward to our toy MichelBaby interpreter</a> and return to this subsection for reference if required, especially when the reader sees us mention <code>bind</code> later.</p>
<p>Still here? Great!</p>
<p>A monad is a <strong>type-constructor</strong>, meaning that given a type <code>t</code> and a monad <code>monad</code>, we have a type <code>t monad</code>.
There are many different monads — e.g. the <a href="https://tezos.gitlab.io/developer/error_monad.html">error monad</a>, the state monad, and the <a href="https://ocsigen.org/lwt/latest/manual/manual"><code>Lwt</code> (lightweight threads) monad</a> — but for the purposes of this blog post, these all just represent actions on some state machine.<sup id="fnref:simplification"><a class="footnote-ref" href="#fn:simplification">5</a></sup>
Thus <code>t monad</code> is a type for computations that compute a <code>t</code> while exerting side-effects on a state monad <code>monad</code>.</p>
<p><a name="monadic-combinators"></a></p>
<p>A monad comes with the following key operations:</p>
<ul>
<li>
<p><code>return : 'a -> 'a monad</code> <br> This inputs a value of type <code>'a</code> and returns that same value, in an empty (or <em>initial</em>) state.</p>
</li>
<li>
<p><code>bind : 'a monad -> ('a -> 'b monad) -> 'b monad</code> <br> This binds a stateful computation from <code>'a</code> to <code>'b</code>, to an input <code>'a</code>-in-state, thus obtaining an output <code>'b</code>-in-state.</p>
</li>
<li>
<p><code>run : 'a monad -> 'a</code> <br> This discards the state and just returns the value.<sup id="fnref:scribble"><a class="footnote-ref" href="#fn:scribble">6</a></sup></p>
</li>
</ul>
<p>C, Java, or Python have monads, or more precisely one monad: a single “global state monad” which represents how the actual state of the computer changes as it executes instructions.
Since in these languages the global state monad permeates all computation, it need not be mentioned explicitly — but it is still there, as you can check: just run a virtual machine, take a snapshot, <em>et voilà</em> your global state is saved to disk as a datum.</p>
<p>OCaml (in common with may other functional programming languages) allows us to be far more fine-grained, by bundling state-fragments up into monads.<sup id="fnref:bytes"><a class="footnote-ref" href="#fn:bytes">7</a></sup>
This has at least two benefits: </p>
<ol>
<li>fragments of machine state are now encapsulated as typed data and can be passed as arguments to other functions (as our crude example above of snapshotting a <span class="caps">VM</span> illustrates); and also, </li>
<li>using monads encourages the programmer to be discriminating about <em>precisely what</em> effects a given computation really needs — if any.</li>
</ol>
<p>So if we think about it, execution of a Michelson script requires a few distinct kinds of state:</p>
<ul>
<li>The state of the blockchain (e.g. to transfer tokens).</li>
<li>The gas level consumed in the current block (especially in case we are about to run out of gas).</li>
<li>A script may interact with the environment via I/O.</li>
<li>A script may also <em>fail</em> because it tried to perform some invalid operation, and this too is interaction with state; namely the <em>failure</em> monad whose effect (if triggered) is to say <em>“I have no return value because I have failed”</em>.</li>
</ul>
<p>In practice the Michelson interpreter uses a monad in OCaml which is the composition of three different monads:
the <a href="https://tezos.gitlab.io/developer/error_monad.html">error monad</a>, the state monad, and the <a href="https://ocsigen.org/lwt/latest/manual/manual"><code>Lwt</code> (lightweight threads) monad</a>.</p>
<p>If the reader sees <code>return</code>, <code>bind</code>, or <code>run</code> below — they are just the monadic glue coming from this or a closely related monadic combination.</p>
<p><a name="interp"></a></p>
<h3 id="an-interpreter-for-michelson-scripts"><a class="toclink" href="#an-interpreter-for-michelson-scripts">An interpreter for Michelson scripts</a></h3>
<p>An interpreter is a program that runs scripts.
Let’s sketch what an OCaml interpreter might look like for a simple object-level language<sup id="fnref:object"><a class="footnote-ref" href="#fn:object">8</a></sup> <strong>MichelBaby</strong>.
MichelBaby is derived from a subset of Michelson’s instructions as used in the currently-live economic protocol Florence; so what follows can be viewed as a simplified but indicative reflection of the smart contracts system as it is currently deployed.</p>
<p><a name="step-1"></a>
Here’s the OCaml type declaration for the abstract datatype <code>instr</code> of MichelBaby instructions (the impatient reader can <a href="#step-2">skip forward to compare with the second version</a>):</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="o">(_,</span> <span class="o">_)</span> <span class="n">instr</span> <span class="o">=</span>
<span class="o">|</span> <span class="nc">Push</span> <span class="o">:</span> <span class="k">'</span><span class="n">b</span> <span class="o">-></span> <span class="o">(</span><span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">b</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">Mul</span> <span class="o">:</span> <span class="o">(</span><span class="n">z</span> <span class="o">*</span> <span class="o">(</span><span class="n">z</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">),</span> <span class="n">z</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">Dec</span> <span class="o">:</span> <span class="o">(</span><span class="n">z</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="n">z</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">CmpNZ</span> <span class="o">:</span> <span class="o">(</span><span class="n">z</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="kt">bool</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">Loop</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="kt">bool</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">-></span> <span class="o">(</span><span class="kt">bool</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">s</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">Dup</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">))</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">Swap</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="o">(</span><span class="k">'</span><span class="n">b</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">),</span> <span class="k">'</span><span class="n">b</span> <span class="o">*</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">))</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">Drop</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">s</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">Dip</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">t</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">-></span> <span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="k">'</span><span class="n">t</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">IO</span> <span class="o">:</span> <span class="o">(</span> <span class="n">z</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="n">z</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">Seq</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">t</span><span class="o">)</span> <span class="n">instr</span> <span class="o">*</span> <span class="o">(</span><span class="k">'</span><span class="n">t</span><span class="o">,</span> <span class="k">'</span><span class="n">u</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">-></span> <span class="o">(</span><span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">u</span><span class="o">)</span> <span class="n">instr</span>
</code></pre></div>
<p><a name="z"></a>
(The type <code>z</code> above is the integer type from the <a href="https://github.com/ocaml/Zarith">standard OCaml library <code>Zarith</code></a> for infinite precision arithmetic.)</p>
<p>We notice that:</p>
<ul>
<li>
<p><code>instr</code> is polymorphic over two type parameters: <code>('s, 't) instr</code>.</p>
<ul>
<li>The first parameter <code>'s</code> represents the <em>input</em> stack type and</li>
<li>the second parameter <code>'t</code> represents the <em>output</em> stack type.</li>
</ul>
</li>
<li>
<p>Each line in the datatype declaration corresponds to an individual instruction.
The type parameters give useful information on the intended meaning.
For example: <a href="mul"></a></p>
<ul>
<li>
<p><code>Mul : (z * (z * 's), z * 's) instr</code> is an instruction that <em>inputs</em> a stack headed by two integers (<code>z * (z * 's)</code>) and <em>outputs</em> a stack headed by one integer (<code>z * 's</code>).
Intuitively, <code>Mul</code> pops the two integers off the stack, multiplies them, and pushes the result.</p>
</li>
<li>
<p><code>Push : 'b -> ('s, 'b * 's) instr</code> is an instruction that inputs a value of type <code>'b</code> and returns an instruction that <em>inputs</em> a stack of type <code>'s</code> and <em>outputs</em> a stack of type <code>'b * 's</code>.
Intuitively, <code>Push b</code> pushes <code>b</code> (of course).</p>
</li>
</ul>
</li>
<li>
<p>There is a special <em>sequence</em> constructor <code>Seq</code> that</p>
<ul>
<li>inputs an instruction that inputs stack <code>'s</code> and outputs a stack <code>'t</code>, and</li>
<li>inputs an instruction that inputs a stack <code>'t</code> and outputs a stack <code>'u</code>, and</li>
<li>returns a composite instruction that inputs a stack <code>'s</code> and outputs a stack <code>'u</code>.</li>
</ul>
<p>Note how the types make it obvious what the interpreter should do with <code>I Seq J</code>: first <code>I</code>, then <code>J</code>.</p>
</li>
<li>
<p><code>IO</code> is a basic <em>effectful</em> instruction — for concreteness, the reader can assume that it writes the topmost integer of the stack to a file, and could be used for example as <a href="#tracing">a hook for logging and profiling execution</a>.</p>
</li>
</ul>
<p><a name="type-checking"></a></p>
<h4 id="michelbaby-typechecking-for-free-courtesy-of-the-ocaml-typechecker"><a class="toclink" href="#michelbaby-typechecking-for-free-courtesy-of-the-ocaml-typechecker">MichelBaby typechecking for free, courtesy of the OCaml typechecker</a></h4>
<p>This style of polymorphic datatype declaration makes the OCaml typechecker act as a MichelBaby typechecker too:
only (representations of) well-typed scripts can inhabit the <code>instr</code> type.
For example, consider the following MichelBaby implemention of <code>fact</code> the factorial:</p>
<p><a name="fact-code"></a></p>
<div class="highlight"><pre><span></span><code><span class="c">(* Inline syntactic sugar for sequencing *)</span>
<span class="k">let</span> <span class="o">(</span> <span class="o">@</span> <span class="o">)</span> <span class="n">s1</span> <span class="n">s2</span> <span class="o">=</span> <span class="nc">Seq</span> <span class="o">(</span><span class="n">s1</span><span class="o">,</span> <span class="n">s2</span><span class="o">)</span>
<span class="c">(* MichelBaby instruction to calculate n! *)</span>
<span class="k">let</span> <span class="n">fact</span> <span class="n">n</span> <span class="o">=</span>
<span class="k">assert</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="o">(</span><span class="n">n</span> <span class="o">></span> <span class="n">zero</span><span class="o">));</span>
<span class="nc">Push</span> <span class="n">n</span>
<span class="o">@</span> <span class="nc">Push</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">of_int</span> <span class="mi">1</span><span class="o">)</span>
<span class="o">@</span> <span class="nc">Dup</span> <span class="o">@</span> <span class="nc">CmpNZ</span> <span class="o">@</span> <span class="nc">Dip</span> <span class="nc">Swap</span>
<span class="o">@</span> <span class="nc">Loop</span> <span class="o">(</span><span class="nc">Dup</span> <span class="o">@</span> <span class="nc">Dip</span> <span class="nc">Swap</span> <span class="o">@</span> <span class="nc">Mul</span> <span class="o">@</span> <span class="nc">IO</span> <span class="o">@</span> <span class="nc">Swap</span> <span class="o">@</span> <span class="nc">Dec</span> <span class="o">@</span> <span class="nc">Dup</span> <span class="o">@</span> <span class="nc">CmpNZ</span><span class="o">)</span>
<span class="o">@</span> <span class="nc">Drop</span>
</code></pre></div>
<p>Hurrah, this is well-typed!
OCaml’s typechecker does not know that <code>fact</code> is a MichelBaby sequence of instructions to compute the factorial, but it does certify that <code>fact</code> has the following type:</p>
<div class="highlight"><pre><span></span><code><span class="n">fact</span> <span class="o">:</span> <span class="n">z</span> <span class="o">-></span> <span class="o">(</span><span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="n">z</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">)</span> <span class="n">instr</span>
</code></pre></div>
<p>In English, this is the OCaml typechecker telling us:</p>
<blockquote>
<p><code>fact</code> will input an integer and return a MichelBaby instruction that inputs a stack and returns a stack of the same shape except it has an integer pushed onto it.</p>
</blockquote>
<p>Now suppose we to forget the <code>CmpNZ</code> instruction above.
Then the OCaml typechecker will reject the definition with the following error message:</p>
<div class="highlight"><pre><span></span><code><span class="nc">This</span> <span class="n">expression</span> <span class="n">has</span> <span class="k">type</span> <span class="o">(</span><span class="kt">bool</span> <span class="o">*</span> <span class="k">'</span><span class="n">a</span><span class="o">,</span> <span class="k">'</span><span class="n">b</span><span class="o">)</span> <span class="n">instr</span>
<span class="n">but</span> <span class="n">an</span> <span class="n">expression</span> <span class="n">was</span> <span class="n">expected</span> <span class="k">of</span> <span class="k">type</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">t</span> <span class="o">*</span> <span class="k">'</span><span class="n">c</span><span class="o">,</span> <span class="k">'</span><span class="n">d</span><span class="o">)</span> <span class="n">instr</span>
<span class="nc">Type</span> <span class="kt">bool</span> <span class="n">is</span> <span class="n">not</span> <span class="n">compatible</span> <span class="k">with</span> <span class="k">type</span> <span class="nn">Z</span><span class="p">.</span><span class="n">t</span>
</code></pre></div>
<p>This is the OCaml typechecker, giving us MichelBaby typechecking for free. </p>
<h4 id="tracing-factorial-a-standard-toy-example"><a class="toclink" href="#tracing-factorial-a-standard-toy-example">Tracing factorial: a standard toy example</a></h4>
<p><a name="tracing"></a></p>
<p><em>Our <a href="#fact-code">implementation <code>fact</code> of factorial</a> is a <strong>tracing factorial</strong>, meaning that it computes factorial and contains an <code>IO</code> <em>tracing</em> (or <em>breakpoint</em>) hook so we can observe intermediate results of the computation using an input/output primitive of our choice — a similar hook is in our <a href="#ocaml-fact">efficient OCaml factorial function below</a>.</p>
<p>Tracing factorial is interesting because it is an easy-to-understand toy example of a mostly pure computation with some side-effects — which is what most smart contracts look like.
This is a convenient toy example: no implication is intended that tracing factorial is the <em>only</em> thing that could be done in MichelBaby or Michelson!</em></p>
<h4 id="the-step-function-version-1"><a class="toclink" href="#the-step-function-version-1">The <code>step</code> function, Version 1</a></h4>
<p>The <code>instr</code> datatype allows us to represent well-typed programs.
Now we need to design an interpreter.
First, suppose we are given a function <code>step</code></p>
<div class="highlight"><pre><span></span><code><span class="n">step</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">i</span><span class="o">,</span> <span class="k">'</span><span class="n">o</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span> <span class="k">'</span><span class="n">i</span> <span class="o">-></span> <span class="k">'</span><span class="n">o</span> <span class="n">monad</span>
</code></pre></div>
<p>Then we can write our <code>interpreter</code> as follows:</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="o">(</span><span class="k">'</span><span class="n">storage</span><span class="o">,</span> <span class="k">'</span><span class="n">argument</span><span class="o">)</span> <span class="n">well_typed_script</span> <span class="o">=</span>
<span class="o">((</span><span class="k">'</span><span class="n">storage</span> <span class="o">*</span> <span class="k">'</span><span class="n">argument</span><span class="o">)</span> <span class="o">*</span> <span class="kt">unit</span><span class="o">,</span> <span class="o">((</span><span class="k">'</span><span class="n">storage</span> <span class="o">*</span> <span class="n">operations</span><span class="o">)</span> <span class="o">*</span> <span class="kt">unit</span><span class="o">))</span> <span class="n">instr</span>
<span class="k">type</span> <span class="o">(</span><span class="k">'</span><span class="n">storage</span><span class="o">,</span> <span class="k">'</span><span class="n">argument</span><span class="o">)</span> <span class="n">interpretation</span> <span class="o">=</span>
<span class="k">'</span><span class="n">storage</span> <span class="o">-></span> <span class="k">'</span><span class="n">argument</span> <span class="o">-></span> <span class="o">(</span><span class="k">'</span><span class="n">storage</span> <span class="o">*</span> <span class="n">operations</span><span class="o">)</span> <span class="n">monad</span>
<span class="k">let</span> <span class="n">interpreter</span> <span class="o">:</span> <span class="k">type</span> <span class="n">storage</span> <span class="n">argument</span><span class="o">.</span>
<span class="o">(</span><span class="n">storage</span><span class="o">,</span> <span class="n">argument</span><span class="o">)</span> <span class="n">well_typed_script</span> <span class="o">-></span> <span class="o">(</span><span class="n">storage</span><span class="o">,</span> <span class="n">argument</span><span class="o">)</span> <span class="n">interpretation</span> <span class="o">=</span>
<span class="k">fun</span> <span class="n">instr</span> <span class="n">storage</span> <span class="n">argument</span> <span class="o">-></span>
<span class="n">bind</span> <span class="o">(</span><span class="n">step</span> <span class="n">instr</span> <span class="o">((</span><span class="n">storage</span><span class="o">,</span> <span class="n">argument</span><span class="o">),</span> <span class="bp">()</span><span class="o">))</span>
<span class="o">@@</span> <span class="k">fun</span> <span class="o">((</span><span class="n">storage</span><span class="o">,</span> <span class="n">operations</span><span class="o">),</span> <span class="bp">()</span><span class="o">)</span> <span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="n">storage</span><span class="o">,</span> <span class="n">operations</span><span class="o">)</span>
</code></pre></div>
<p><a name="version-1"></a></p>
<p>So now we just have to write code for <code>step</code>.
Standard practice is to just follow the inductive structure of <code>instr</code>.</p>
<p>We present <strong>Version 1</strong> of the <code>step</code> function (cf. Versions <a href="#version-1.1">1.1</a> and <a href="#version-2">2</a> below):</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="k">rec</span> <span class="n">step</span> <span class="o">:</span> <span class="k">type</span> <span class="n">a</span> <span class="n">b</span><span class="o">.</span> <span class="o">(</span><span class="n">a</span><span class="o">,</span> <span class="n">b</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="n">b</span> <span class="n">monad</span> <span class="o">=</span>
<span class="k">fun</span> <span class="n">instr</span> <span class="n">stack</span> <span class="o">-></span>
<span class="k">match</span> <span class="o">(</span><span class="n">instr</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span> <span class="k">with</span>
<span class="o">|</span> <span class="nc">Push</span> <span class="n">x</span><span class="o">,</span> <span class="n">stack</span>
<span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">Mul</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="o">(</span><span class="n">y</span><span class="o">,</span> <span class="n">stack</span><span class="o">))</span>
<span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">mul</span> <span class="n">x</span> <span class="n">y</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">Dec</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="o">(</span><span class="n">sub</span> <span class="n">x</span> <span class="o">(</span><span class="n">of_int</span> <span class="mi">1</span><span class="o">)),</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">CmpNZ</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="o">(</span><span class="n">compare</span> <span class="n">x</span> <span class="n">zero</span> <span class="o"><></span> <span class="mi">0</span><span class="o">),</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">Loop</span> <span class="o">_,</span> <span class="o">(</span><span class="bp">false</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">-></span> <span class="n">return</span> <span class="n">stack</span>
<span class="o">|</span> <span class="nc">Loop</span> <span class="n">body</span><span class="o">,</span> <span class="o">(</span><span class="bp">true</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">-></span> <span class="n">bind</span> <span class="o">(</span><span class="n">step</span> <span class="n">body</span> <span class="n">stack</span><span class="o">)</span> <span class="o">@@</span> <span class="k">fun</span> <span class="n">stack</span> <span class="o">-></span> <span class="n">step</span> <span class="o">(</span><span class="nc">Loop</span> <span class="n">body</span><span class="o">)</span> <span class="n">stack</span>
<span class="o">|</span> <span class="nc">Dup</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">))</span>
<span class="o">|</span> <span class="nc">Swap</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="o">(</span><span class="n">y</span><span class="o">,</span> <span class="n">stack</span><span class="o">))</span>
<span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="n">y</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">))</span>
<span class="o">|</span> <span class="nc">Drop</span><span class="o">,</span> <span class="o">(_,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">-></span> <span class="n">return</span> <span class="n">stack</span>
<span class="o">|</span> <span class="nc">Dip</span> <span class="n">i</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">-></span> <span class="n">bind</span> <span class="o">(</span><span class="n">step</span> <span class="n">i</span> <span class="n">stack</span><span class="o">)</span> <span class="o">@@</span> <span class="k">fun</span> <span class="n">stack</span> <span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">IO</span><span class="o">,</span> <span class="o">(</span><span class="n">z</span><span class="o">,</span> <span class="o">_)</span>
<span class="o">-></span> <span class="n">bind</span> <span class="o">(</span><span class="n">io</span> <span class="n">z</span><span class="o">)</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-></span> <span class="n">return</span> <span class="n">stack</span>
<span class="o">|</span> <span class="nc">Seq</span> <span class="o">(</span><span class="n">i1</span><span class="o">,</span> <span class="n">i2</span><span class="o">),</span> <span class="n">stack</span>
<span class="o">-></span> <span class="n">bind</span> <span class="o">(</span><span class="n">step</span> <span class="n">i1</span> <span class="n">stack</span><span class="o">)</span> <span class="o">@@</span> <span class="k">fun</span> <span class="n">stack</span> <span class="o">-></span> <span class="n">step</span> <span class="n">i2</span> <span class="n">stack</span>
</code></pre></div>
<p>The recursive function <code>step</code> observes the instruction and the stack and
produces a monadic computation using <code>return</code> and <code>bind</code>.
Let’s read through some cases of this definition:</p>
<ul>
<li>
<p><em>Case for <code>Push</code>.</em>
As prescribed by the definition of <code>Push</code>,
the input stack type is <code>'s</code> and
we must return a stack of the form <code>'b * 's</code>.
That is the type of <code>(x, stack)</code>.
Besides, since we need the returned value to be of type <code>('b * 's) monad</code>,
we use <code>return</code> to turn <code>(x, stack)</code> into a monad computation.</p>
</li>
<li>
<p><em>Case for <code>Seq</code>.</em>
As prescribed by the definition of <code>Seq</code>,
the instruction <code>i1</code> must be of type <code>('s, 't) instr</code> and
<code>i2</code> of type <code>('t, 'u) instr</code>.
The recursive call <code>(step i1 stack)</code> is well-typed and
returns a value of type <code>'t monad</code>.
Using <code>bind</code>, we can retrieve the stack of type <code>'t</code> needed to interpret <code>i2</code>.
The final computation has type <code>'u monad</code> as expected.</p>
</li>
</ul>
<p><a name="io-interp"></a></p>
<ul>
<li><em>Case for <code>IO</code>.</em>
Here, we use an effectful operation <code>io : z -> unit monad</code> and
compose it with a computation that returns that stack unchanged.</li>
</ul>
<p>We check that the interpreter correctly computes the factorial on an input (other inputs work too!):</p>
<div class="highlight"><pre><span></span><code><span class="o">#</span> <span class="n">run</span> <span class="o">(</span><span class="n">step</span> <span class="o">(</span><span class="n">fact</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">of_int</span> <span class="mi">100</span><span class="o">))</span> <span class="bp">()</span><span class="o">)</span> <span class="o">|></span> <span class="n">fst</span> <span class="o">=</span> <span class="n">ocaml_fact</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">of_int</span> <span class="mi">100</span><span class="o">);;</span>
<span class="o">-</span> <span class="o">:</span> <span class="kt">bool</span> <span class="o">=</span> <span class="bp">true</span>
</code></pre></div>
<p>This interpreter has a good property: it can’t fail.
Because <a href="#type-checking">we get MichelBaby type-checking for free</a>, <em>every</em> application of <code>step</code> to a well-typed argument in <code>instr</code> is guaranteed to successfully execute (provided the ambient execution environment doesn’t suffer an overflow; see next paragraph).
We can’t fail on an ill-formed operation like multiplying two strings or popping an empty stack, because our free MichelBaby typechecker will detect and reject the instruction-sequence as ill-typed. </p>
<p>Unfortunately, our interpreter above is not yet “right”, because <code>step</code> is not <a href="https://en.wikipedia.org/wiki/Tail_call">tail recursive</a>.
Note that:</p>
<ul>
<li>Recursive calls to <code>step</code> are not the final step in the computation in the clauses for <code>Seq</code> (in <code>step i1 stack</code>) and <code>Dip</code> (in <code>step i stack</code>).</li>
<li>For comparison, the calls in <code>Loop</code> to <code>step (Loop body) stack</code> and in <code>Seq</code> to <code>step i2 stack</code>, <em>are</em> tail recursive.</li>
</ul>
<p>This is not “right” because each call to the interpretation loop that passes through a non-tail-recursive call to <code>step</code>, may consume a bit of the OCaml calling stack,
and we can use this to provoke incorrect behaviour, namely a <a href="https://en.wikipedia.org/wiki/Stack_overflow">stack overflow</a> (when the call stack fills up and the computer has to terminate execution because it runs out of memory):</p>
<div class="highlight"><pre><span></span><code><span class="o">#</span> <span class="k">let</span> <span class="k">rec</span> <span class="n">long_seq</span> <span class="n">accu</span> <span class="n">n</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">n</span> <span class="o">=</span> <span class="mi">0</span> <span class="k">then</span> <span class="n">accu</span>
<span class="k">else</span> <span class="n">long_seq</span> <span class="o">(</span><span class="nc">Seq</span> <span class="o">(</span><span class="n">accu</span><span class="o">,</span> <span class="nc">Seq</span> <span class="o">(</span><span class="nc">Push</span> <span class="nn">Z</span><span class="p">.</span><span class="n">zero</span><span class="o">,</span> <span class="nc">Drop</span><span class="o">)))</span> <span class="o">(</span><span class="n">n</span> <span class="o">-</span> <span class="mi">1</span><span class="o">);;</span>
<span class="k">val</span> <span class="n">long_seq</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span><span class="o">,</span> <span class="k">'</span><span class="n">b</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span> <span class="kt">int</span> <span class="o">-></span> <span class="o">(</span><span class="k">'</span><span class="n">a</span><span class="o">,</span> <span class="k">'</span><span class="n">b</span><span class="o">)</span> <span class="n">instr</span> <span class="o">=</span> <span class="o"><</span><span class="k">fun</span><span class="o">></span>
<span class="o">#</span> <span class="n">step</span> <span class="o">(</span><span class="n">long_seq</span> <span class="o">(</span><span class="nc">Push</span> <span class="nn">Z</span><span class="p">.</span><span class="n">zero</span><span class="o">)</span> <span class="mi">10000000</span><span class="o">)</span> <span class="bp">()</span><span class="o">;;</span>
<span class="nc">Stack</span> <span class="n">overflow</span> <span class="n">during</span> <span class="n">evaluation</span> <span class="o">(</span><span class="n">looping</span> <span class="n">recursion</span><span class="o">?).</span>
</code></pre></div>
<p><em>When</em> the stack overflow error occurs, is architecture-dependent and depends on how much stack is available to consume.
A cleaner way to handle this is to count the nesting depth of non-tail-recursive calls, and abort if we go too far.<sup id="fnref:python"><a class="footnote-ref" href="#fn:python">9</a></sup></p>
<p><a name="version-1.1"></a></p>
<h4 id="the-step-function-version-11"><a class="toclink" href="#the-step-function-version-11">The <code>step</code> function, Version 1.1</a></h4>
<p>We add a <code>depth</code> counter to <a href="#version-1">Version 1 of <code>step</code></a> (cf. <a href="#version-2">Version 2</a> below):</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="k">rec</span> <span class="n">step</span> <span class="o">:</span> <span class="k">type</span> <span class="n">a</span> <span class="n">b</span><span class="o">.</span> <span class="kt">int</span> <span class="o">-></span> <span class="o">(</span><span class="n">a</span><span class="o">,</span> <span class="n">b</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="n">b</span> <span class="n">monad</span> <span class="o">=</span>
<span class="k">fun</span> <span class="n">depth</span> <span class="n">instr</span> <span class="n">stack</span> <span class="o">-></span>
<span class="k">if</span> <span class="n">depth</span> <span class="o">></span> <span class="mi">100000</span> <span class="k">then</span> <span class="n">fail</span> <span class="s2">"Too many recursion"</span>
<span class="k">else</span> <span class="k">match</span> <span class="o">(</span><span class="n">instr</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span> <span class="k">with</span>
<span class="o">|</span> <span class="nc">Push</span> <span class="n">x</span><span class="o">,</span> <span class="n">stack</span>
<span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">Mul</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="o">(</span><span class="n">y</span><span class="o">,</span> <span class="n">stack</span><span class="o">))</span>
<span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">mul</span> <span class="n">x</span> <span class="n">y</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">Dec</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="o">(</span><span class="n">sub</span> <span class="n">x</span> <span class="o">(</span><span class="n">of_int</span> <span class="mi">1</span><span class="o">)),</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">CmpNZ</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="o">(</span><span class="n">compare</span> <span class="n">x</span> <span class="n">zero</span> <span class="o"><></span> <span class="mi">0</span><span class="o">),</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">Loop</span> <span class="o">_,</span> <span class="o">(</span><span class="bp">false</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">-></span> <span class="n">return</span> <span class="n">stack</span>
<span class="o">|</span> <span class="nc">Loop</span> <span class="n">body</span><span class="o">,</span> <span class="o">(</span><span class="bp">true</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">-></span> <span class="n">bind</span> <span class="o">(</span><span class="n">step</span> <span class="o">(</span><span class="n">depth</span> <span class="o">+</span> <span class="mi">1</span><span class="o">)</span> <span class="n">body</span> <span class="n">stack</span><span class="o">)</span> <span class="o">@@</span> <span class="k">fun</span> <span class="n">stack</span> <span class="o">-></span> <span class="n">step</span> <span class="n">depth</span> <span class="o">(</span><span class="nc">Loop</span> <span class="n">body</span><span class="o">)</span> <span class="n">stack</span>
<span class="o">|</span> <span class="nc">Seq</span> <span class="o">(</span><span class="n">i1</span><span class="o">,</span> <span class="n">i2</span><span class="o">),</span> <span class="n">stack</span>
<span class="o">-></span> <span class="n">bind</span> <span class="o">(</span><span class="n">step</span> <span class="o">(</span><span class="n">depth</span> <span class="o">+</span> <span class="mi">1</span><span class="o">)</span> <span class="n">i1</span> <span class="n">stack</span><span class="o">)</span> <span class="o">@@</span> <span class="k">fun</span> <span class="n">stack</span> <span class="o">-></span> <span class="n">step</span> <span class="n">depth</span> <span class="n">i2</span> <span class="n">stack</span>
<span class="o">|</span> <span class="nc">Dup</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">))</span>
<span class="o">|</span> <span class="nc">Swap</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="o">(</span><span class="n">y</span><span class="o">,</span> <span class="n">stack</span><span class="o">))</span>
<span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="n">y</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">))</span>
<span class="o">|</span> <span class="nc">Drop</span><span class="o">,</span> <span class="o">(_,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">-></span> <span class="n">return</span> <span class="n">stack</span>
<span class="o">|</span> <span class="nc">Dip</span> <span class="n">i</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">-></span> <span class="n">bind</span> <span class="o">(</span><span class="n">step</span> <span class="o">(</span><span class="n">depth</span> <span class="o">+</span> <span class="mi">1</span><span class="o">)</span> <span class="n">i</span> <span class="n">stack</span><span class="o">)</span> <span class="o">@@</span> <span class="k">fun</span> <span class="n">stack</span> <span class="o">-></span> <span class="n">return</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">stack</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">IO</span><span class="o">,</span> <span class="o">(</span><span class="n">z</span><span class="o">,</span> <span class="o">_)</span>
<span class="o">-></span> <span class="n">bind</span> <span class="o">(</span><span class="n">io</span> <span class="n">z</span><span class="o">)</span> <span class="o">@@</span> <span class="k">fun</span> <span class="bp">()</span> <span class="o">-></span> <span class="n">return</span> <span class="n">stack</span>
</code></pre></div>
<p>The program is still non-tail-recursive but at least we have replaced a stack overflow which we cannot control, with an explicit branch within our own program which we can control.</p>
<p>Some tests are required to determine whether <code>100000</code> is a good limit.<sup id="fnref:limit"><a class="footnote-ref" href="#fn:limit">10</a></sup>
Once this limit value is appropriately chosen, we can claim to have made the interpreter “right”:</p>
<ul>
<li>it follows a standard OCaml programming style (hence natural to reason about), and</li>
<li>it is robust to stack overflows.</li>
</ul>
<h3 id="making-the-interpreter-fast"><a class="toclink" href="#making-the-interpreter-fast">Making the interpreter fast</a></h3>
<p>How fast is our <code>step</code> function?
To get an idea, we can compare how long it takes to compute the factorial of a hundred — <code>100! = 100*99*98*...*1</code> — with an equivalent native OCaml implementation of the factorial:</p>
<p><a name="ocaml-fact"></a></p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="n">lwt_fact</span> <span class="n">n</span> <span class="o">=</span>
<span class="k">let</span> <span class="k">rec</span> <span class="n">aux</span> <span class="n">k</span> <span class="n">accu</span> <span class="o">=</span>
<span class="k">if</span> <span class="nn">Z</span><span class="p">.</span><span class="o">(</span><span class="n">compare</span> <span class="n">k</span> <span class="n">zero</span> <span class="o">=</span> <span class="mi">0</span><span class="o">)</span> <span class="k">then</span> <span class="n">return</span> <span class="n">accu</span>
<span class="k">else</span>
<span class="k">let</span> <span class="n">accu</span> <span class="o">=</span> <span class="nn">Z</span><span class="p">.</span><span class="n">mul</span> <span class="n">accu</span> <span class="n">k</span> <span class="k">in</span>
<span class="n">bind</span> <span class="o">(</span><span class="n">io</span> <span class="n">accu</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="bp">()</span> <span class="o">-></span> <span class="n">aux</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">sub</span> <span class="n">k</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">of_int</span> <span class="mi">1</span><span class="o">))</span> <span class="n">accu</span><span class="o">)</span>
<span class="k">in</span>
<span class="n">aux</span> <span class="n">n</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">of_int</span> <span class="mi">1</span><span class="o">)</span>
</code></pre></div>
<p>Here are the benchmarks, for the <a href="#ocaml-fact">OCaml factorial function</a> and <a href="#version-1.1">V1.1 of our <code>step</code> function</a>:</p>
<div class="highlight"><pre><span></span><code>┌─────────────┬──────────┬────────────┐
│ Name │ Time/Run │ Percentage │
├─────────────┼──────────┼────────────┤
│ OCaml │ 3.71us │ 23.4% │
│ Step (V1.1) │ 15.83us │ 100.0% │
└─────────────┴──────────┴────────────┘
</code></pre></div>
<p>On this specific example,
the interpreter is five times slower than the reference implementation in OCaml.
Why?
Profiling execution with <a href="https://web.archive.org/web/20210618120941/https://www.brendangregg.com/perf.html">the linux <code>perf</code> command</a>, we learn that 50% of the time is spent in the <a href="#monadic-combinators">monadic combinators (mentioned above)</a>.
Mostly this is due to the code for handling <code>Seq</code> in <a href="#version-1.1">the code above</a>, in which <code>bind</code> is called
to glue together the interpretation of the two instructions of a sequence:
this has a negative impact on the performance
because <code>bind</code> happens to be a rather complex operation when the monad includes the <a href="https://ocsigen.org/lwt/latest/manual/manual"><code>Lwt</code> (lightweight threads) monad</a>.</p>
<p>We need to get rid of those <code>bind</code>s.</p>
<h4 id="reducing-the-number-of-binds"><a class="toclink" href="#reducing-the-number-of-binds">Reducing the number of <code>bind</code>s</a></h4>
<p>Let’s reexamine the <a href="#fact-code">MichelBaby code for <code>fact</code></a> above:</p>
<div class="highlight"><pre><span></span><code><span class="c">(* Inline syntactic sugar for sequencing *)</span>
<span class="k">let</span> <span class="o">(</span> <span class="o">@</span> <span class="o">)</span> <span class="n">s1</span> <span class="n">s2</span> <span class="o">=</span> <span class="nc">Seq</span> <span class="o">(</span><span class="n">s1</span><span class="o">,</span> <span class="n">s2</span><span class="o">)</span>
<span class="c">(* MichelBaby instruction to calculate n! *)</span>
<span class="k">let</span> <span class="n">fact</span> <span class="n">n</span> <span class="o">=</span>
<span class="k">assert</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="o">(</span><span class="n">n</span> <span class="o">></span> <span class="n">zero</span><span class="o">));</span>
<span class="nc">Push</span> <span class="n">n</span>
<span class="o">@</span> <span class="nc">Push</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">of_int</span> <span class="mi">1</span><span class="o">)</span>
<span class="o">@</span> <span class="nc">Dup</span> <span class="o">@</span> <span class="nc">CmpNZ</span> <span class="o">@</span> <span class="nc">Dip</span> <span class="nc">Swap</span>
<span class="o">@</span> <span class="nc">Loop</span> <span class="o">(</span><span class="nc">Dup</span> <span class="o">@</span> <span class="nc">Dip</span> <span class="nc">Swap</span> <span class="o">@</span> <span class="nc">Mul</span> <span class="o">@</span> <span class="nc">IO</span> <span class="o">@</span> <span class="nc">Swap</span> <span class="o">@</span> <span class="nc">Dec</span> <span class="o">@</span> <span class="nc">Dup</span> <span class="o">@</span> <span class="nc">CmpNZ</span><span class="o">)</span>
<span class="o">@</span> <span class="nc">Drop</span>
</code></pre></div>
<p><a name="cost"></a></p>
<p>The MichelBaby script contains only one inherently effectful instruction: <code>IO</code>.
Yet each <code>Seq</code> (written <code>@</code> above) calls a monadic <code>bind</code> operation, which is expensive.
This means that we pay the cost of the monadic abstraction repeatedly, when in fact a pure computation could perform (all but one) of the calculations more efficiently.</p>
<p>Can we separate the pure instructions from the impure ones and then use <code>bind</code> just when impure instructions enter the scene?
We can start by syntactically separating the pure computations from the impure ones,
taking inspiration from our <a href="#ocaml-fact">OCaml version of the factorial</a>:</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="n">lwt_fact</span> <span class="n">n</span> <span class="o">=</span>
<span class="k">let</span> <span class="k">rec</span> <span class="n">aux</span> <span class="n">k</span> <span class="n">accu</span> <span class="o">=</span>
<span class="k">if</span> <span class="nn">Z</span><span class="p">.</span><span class="o">(</span><span class="n">compare</span> <span class="n">k</span> <span class="n">zero</span> <span class="o">=</span> <span class="mi">0</span><span class="o">)</span> <span class="k">then</span> <span class="n">return</span> <span class="n">accu</span>
<span class="k">else</span>
<span class="k">let</span> <span class="n">accu</span> <span class="o">=</span> <span class="nn">Z</span><span class="p">.</span><span class="n">mul</span> <span class="n">accu</span> <span class="n">k</span> <span class="k">in</span>
<span class="n">bind</span> <span class="o">(</span><span class="n">io</span> <span class="n">accu</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="bp">()</span> <span class="o">-></span> <span class="n">aux</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">sub</span> <span class="n">k</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">of_int</span> <span class="mi">1</span><span class="o">))</span> <span class="n">accu</span><span class="o">)</span>
<span class="k">in</span>
<span class="n">aux</span> <span class="n">n</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">of_int</span> <span class="mi">1</span><span class="o">)</span>
</code></pre></div>
<p>Notice how</p>
<ul>
<li>the accumulator <code>accu</code> accumulates the result of a sequence of multiplications — that’s the pure computation, whereas</li>
<li>the impure computation — namely the call to <a href="#io-interp">the effectful operation <code>io</code></a> — is performed in the body of the <code>aux</code> function.</li>
</ul>
<p>We will now rephrase <code>step</code> so that:</p>
<ol>
<li>
<p>The pure computations rewrite the input stack passed as an argument to a tail recursive call to the next instruction.</p>
</li>
<li>
<p>The impure computations return monadic computations.</p>
</li>
</ol>
<p><a name="step-2"></a>
First, we must rephrase the datatype that defines the instructions, so that every instruction points to the next instruction (compare the code below with <a href="#step-1">the first version</a>, and <a href="#z"><code>z</code> explained here</a>):</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="o">(_,</span> <span class="o">_)</span> <span class="n">instr</span> <span class="o">=</span>
<span class="o">|</span> <span class="nc">KHalt</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">s</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">KPush</span> <span class="o">:</span> <span class="k">'</span><span class="n">b</span> <span class="o">*</span> <span class="o">(</span><span class="k">'</span><span class="n">b</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span>
<span class="o">(</span><span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">KMul</span> <span class="o">:</span> <span class="o">(</span><span class="n">z</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span>
<span class="o">(</span><span class="n">z</span> <span class="o">*</span> <span class="o">(</span><span class="n">z</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">),</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">KDec</span> <span class="o">:</span> <span class="o">(</span><span class="n">z</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span>
<span class="o">(</span><span class="n">z</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">KCmpNZ</span> <span class="o">:</span> <span class="o">(</span><span class="kt">bool</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span>
<span class="o">(</span><span class="n">z</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">KLoop</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="kt">bool</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">)</span> <span class="n">instr</span> <span class="o">*</span> <span class="o">(</span><span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span>
<span class="o">(</span><span class="kt">bool</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">KDup</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">),</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span>
<span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span> <span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">KSwap</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="o">(</span><span class="k">'</span><span class="n">b</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">),</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span>
<span class="o">(</span><span class="k">'</span><span class="n">b</span> <span class="o">*</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">),</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">KDrop</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span>
<span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">KDip</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">t</span><span class="o">,</span> <span class="k">'</span><span class="n">s</span><span class="o">)</span> <span class="n">instr</span> <span class="o">*</span> <span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span>
<span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="o">*</span> <span class="k">'</span><span class="n">t</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span>
<span class="o">|</span> <span class="nc">KIO</span> <span class="o">:</span> <span class="o">(</span><span class="n">z</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span>
<span class="o">(</span><span class="n">z</span> <span class="o">*</span> <span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">f</span><span class="o">)</span> <span class="n">instr</span><span class="o">`</span>
</code></pre></div>
<p>Let’s compare <a href="#mul"><code>Mul</code> from the first version</a> with <code>KMul</code> here:</p>
<ul>
<li>
<p><code>Mul : (z * (z * 's), z * 's) instr</code> is an instruction that </p>
<p>*<em>inputs</em> a stack headed by two integers (<code>z * (z * 's)</code>) and
* <em>outputs</em> a stack headed by one integer (<code>z * 's</code>).</p>
<p>Intuitively, <code>Mul</code> pulls the two integers off the stack, multiplies them, and pushes the result.</p>
</li>
<li>
<p><code>KMul : (z * 's, 'f) instr -> (z * (z * 's), 'f) instr</code> is an instruction that </p>
<ul>
<li><em>inputs</em> a sequence of instructions that inputs a stack headed by an integer <code>z * 's</code> and outputs a stack <code>'f</code>, and </li>
<li><em>outputs</em> a sequence of instructions that inputs a stack headed by two integers <code>z * (z * 's)</code> and outputs a stack <code>'f</code>.</li>
</ul>
</li>
</ul>
<p>Intuitively, <code>KMul</code> just adds <code>Mul</code> onto the head of an existing instruction sequence.</p>
<ul>
<li>Our new datatype does not represent <em>instructions</em>; it represents <em>instruction-sequences</em>.</li>
<li>The datatype-constructors represent instructions to push instructions onto sequences.</li>
</ul>
<p><em>Ever written a shopping list?
The family sits around the table calling out household essentials, and then Dad (or Mum — whoever does the shopping) adds items to a written list.
<code>KMul</code> corresponds to somebody saying “Don’t forget to buy multiplication” and then Dad (or Mum) writes “Mul-ti-pli-cation” on the end of the list.</p>
<p>Except that, because this blog post is about Very Serious Programming, our lists start on the right-hand-side and expand leftwards.</p>
<p>The issue with <a href="#cost">the cost of <code>bind</code></a> is also relevant in this analogy.
Most of the time it doesn’t matter what order items are put into the shopping basket.
So it makes sense to split the shopping list into a small number of big heavy things that need put in the trolley last, and a large number of smaller things that can be efficiently executed by rapidly traversing the supermarket in whatever order is most efficient for that supermarket’s layout.
Isn’t computer science wonderful?</em></p>
<p><code>KHalt</code> is a new instruction, to represent the end of our to-do list (think: <span class="caps">EOF</span> or <span class="caps">EOS</span> marker).</p>
<p>There is one catch to our to-do list / shopping list analogy (to be fair, the reader can expect programming smart contracts to be a <em>little</em> harder than shopping):
it may not always be possible to determine the next instruction entirely in advance.
Execution may dynamically depend on input parameters:
for example, a control-flow operator like <code>KLoop</code> dynamically chooses the next instruction by observing a Boolean.</p>
<p><a href="control-stack"></a></p>
<p>This means that we need another stack — a <em>control stack</em> —
that will allow control-flow operators to dynamically define what the next instruction should be:</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="o">(_,</span> <span class="o">_)</span> <span class="n">instrs</span> <span class="o">=</span>
<span class="o">|</span> <span class="nc">KNil</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">s</span><span class="o">)</span> <span class="n">instrs</span>
<span class="o">|</span> <span class="nc">KCons</span> <span class="o">:</span> <span class="o">(</span><span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">t</span><span class="o">)</span> <span class="n">instr</span> <span class="o">*</span> <span class="o">(</span><span class="k">'</span><span class="n">t</span><span class="o">,</span> <span class="k">'</span><span class="n">u</span><span class="o">)</span> <span class="n">instrs</span> <span class="o">-></span> <span class="o">(</span><span class="k">'</span><span class="n">s</span><span class="o">,</span> <span class="k">'</span><span class="n">u</span><span class="o">)</span> <span class="n">instrs</span>
</code></pre></div>
<p>Intuitively, if <code>instr</code> represents a sequence of instructions then <code>instrs</code> represents a list of sequences of instructions, with the constraint that the final stack type (<code>'t</code> above) of each sequence of instructions on the list, must be compatible with the input stack type (<code>'t</code> above) of the next sequence in the list (if any).</p>
<h4 id="the-step-function-version-2"><a class="toclink" href="#the-step-function-version-2">The <code>step</code> function, Version 2</a></h4>
<p><a name="version-2"></a></p>
<p>We present <strong>Version 2</strong> of the <code>step</code> function (cf Versions <a href="#version-1">1</a> and <a href="#version-1.1">1.1</a> above):</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="n">step</span> <span class="o">:</span> <span class="k">type</span> <span class="n">a</span> <span class="n">b</span> <span class="n">p</span><span class="o">.</span> <span class="o">(</span><span class="n">a</span><span class="o">,</span> <span class="n">b</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="n">b</span> <span class="n">monad</span> <span class="o">=</span>
<span class="k">fun</span> <span class="n">i</span> <span class="n">stack</span> <span class="o">-></span>
<span class="k">let</span> <span class="k">rec</span> <span class="n">exec</span> <span class="o">:</span> <span class="k">type</span> <span class="n">a</span> <span class="n">i</span> <span class="n">p</span><span class="o">.</span> <span class="o">(</span><span class="n">a</span><span class="o">,</span> <span class="n">i</span><span class="o">)</span> <span class="n">instr</span> <span class="o">-></span> <span class="o">(</span><span class="n">i</span><span class="o">,</span> <span class="n">b</span><span class="o">)</span> <span class="n">instrs</span> <span class="o">-></span> <span class="n">a</span> <span class="o">-></span> <span class="n">b</span> <span class="n">monad</span> <span class="o">=</span>
<span class="k">fun</span> <span class="n">k</span> <span class="n">ks</span> <span class="n">s</span> <span class="o">-></span>
<span class="k">match</span> <span class="o">(</span><span class="n">k</span><span class="o">,</span> <span class="n">ks</span><span class="o">)</span> <span class="k">with</span>
<span class="o">|</span> <span class="nc">KHalt</span><span class="o">,</span> <span class="nc">KNil</span> <span class="o">-></span> <span class="n">return</span> <span class="n">s</span>
<span class="o">|</span> <span class="nc">KHalt</span><span class="o">,</span> <span class="nc">KCons</span> <span class="o">(</span><span class="n">k</span><span class="o">,</span> <span class="n">ks</span><span class="o">)</span> <span class="o">-></span> <span class="n">exec</span> <span class="n">k</span> <span class="n">ks</span> <span class="n">s</span>
<span class="o">|</span> <span class="nc">KIO</span> <span class="n">k</span><span class="o">,</span> <span class="n">ks</span> <span class="o">-></span>
<span class="k">let</span> <span class="n">z</span><span class="o">,</span> <span class="o">_</span> <span class="o">=</span> <span class="n">s</span> <span class="k">in</span>
<span class="n">bind</span> <span class="o">(</span><span class="n">io</span> <span class="n">z</span><span class="o">)</span> <span class="o">(</span><span class="k">fun</span> <span class="bp">()</span> <span class="o">-></span> <span class="n">exec</span> <span class="n">k</span> <span class="n">ks</span> <span class="n">s</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">KPush</span> <span class="o">(</span><span class="n">z</span><span class="o">,</span> <span class="n">k</span><span class="o">),</span> <span class="n">ks</span> <span class="o">-></span> <span class="n">exec</span> <span class="n">k</span> <span class="n">ks</span> <span class="o">(</span><span class="n">z</span><span class="o">,</span> <span class="n">s</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">KLoop</span> <span class="o">(</span><span class="n">ki</span><span class="o">,</span> <span class="n">k</span><span class="o">),</span> <span class="n">ks</span> <span class="o">-></span> <span class="o">(</span>
<span class="k">match</span> <span class="n">s</span> <span class="k">with</span>
<span class="o">|</span> <span class="bp">true</span><span class="o">,</span> <span class="n">s</span> <span class="o">-></span> <span class="n">exec</span> <span class="n">ki</span> <span class="o">(</span><span class="nc">KCons</span> <span class="o">(</span><span class="nc">KLoop</span> <span class="o">(</span><span class="n">ki</span><span class="o">,</span> <span class="n">k</span><span class="o">),</span> <span class="n">ks</span><span class="o">))</span> <span class="n">s</span>
<span class="o">|</span> <span class="bp">false</span><span class="o">,</span> <span class="n">s</span> <span class="o">-></span> <span class="n">exec</span> <span class="n">k</span> <span class="n">ks</span> <span class="n">s</span> <span class="o">)</span>
<span class="o">|</span> <span class="nc">KMul</span> <span class="n">k</span><span class="o">,</span> <span class="n">ks</span> <span class="o">-></span>
<span class="k">let</span> <span class="n">x</span><span class="o">,</span> <span class="o">(</span><span class="n">y</span><span class="o">,</span> <span class="n">s</span><span class="o">)</span> <span class="o">=</span> <span class="n">s</span> <span class="k">in</span>
<span class="n">exec</span> <span class="n">k</span> <span class="n">ks</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">mul</span> <span class="n">x</span> <span class="n">y</span><span class="o">,</span> <span class="n">s</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">KDec</span> <span class="n">k</span><span class="o">,</span> <span class="n">ks</span> <span class="o">-></span>
<span class="k">let</span> <span class="n">x</span><span class="o">,</span> <span class="n">s</span> <span class="o">=</span> <span class="n">s</span> <span class="k">in</span>
<span class="n">exec</span> <span class="n">k</span> <span class="n">ks</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">sub</span> <span class="n">x</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="n">of_int</span> <span class="mi">1</span><span class="o">),</span> <span class="n">s</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">KCmpNZ</span> <span class="n">k</span><span class="o">,</span> <span class="n">ks</span> <span class="o">-></span>
<span class="k">let</span> <span class="n">x</span><span class="o">,</span> <span class="n">s</span> <span class="o">=</span> <span class="n">s</span> <span class="k">in</span>
<span class="n">exec</span> <span class="n">k</span> <span class="n">ks</span> <span class="o">(</span><span class="nn">Z</span><span class="p">.</span><span class="o">(</span><span class="n">compare</span> <span class="n">x</span> <span class="n">zero</span><span class="o">)</span> <span class="o"><></span> <span class="mi">0</span><span class="o">,</span> <span class="n">s</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">KDup</span> <span class="n">k</span><span class="o">,</span> <span class="n">ks</span> <span class="o">-></span>
<span class="k">let</span> <span class="n">x</span><span class="o">,</span> <span class="n">s</span> <span class="o">=</span> <span class="n">s</span> <span class="k">in</span>
<span class="n">exec</span> <span class="n">k</span> <span class="n">ks</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">s</span><span class="o">))</span>
<span class="o">|</span> <span class="nc">KSwap</span> <span class="n">k</span><span class="o">,</span> <span class="n">ks</span> <span class="o">-></span>
<span class="k">let</span> <span class="n">x</span><span class="o">,</span> <span class="o">(</span><span class="n">y</span><span class="o">,</span> <span class="n">s</span><span class="o">)</span> <span class="o">=</span> <span class="n">s</span> <span class="k">in</span>
<span class="n">exec</span> <span class="n">k</span> <span class="n">ks</span> <span class="o">(</span><span class="n">y</span><span class="o">,</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">s</span><span class="o">))</span>
<span class="o">|</span> <span class="nc">KDrop</span> <span class="n">k</span><span class="o">,</span> <span class="n">ks</span> <span class="o">-></span>
<span class="k">let</span> <span class="o">_,</span> <span class="n">s</span> <span class="o">=</span> <span class="n">s</span> <span class="k">in</span>
<span class="n">exec</span> <span class="n">k</span> <span class="n">ks</span> <span class="n">s</span>
<span class="o">|</span> <span class="nc">KDip</span> <span class="o">(</span><span class="n">ki</span><span class="o">,</span> <span class="n">k</span><span class="o">),</span> <span class="n">ks</span> <span class="o">-></span>
<span class="k">let</span> <span class="n">x</span><span class="o">,</span> <span class="n">s</span> <span class="o">=</span> <span class="n">s</span> <span class="k">in</span>
<span class="n">exec</span> <span class="n">ki</span> <span class="o">(</span><span class="nc">KCons</span> <span class="o">(</span><span class="nc">KPush</span> <span class="o">(</span><span class="n">x</span><span class="o">,</span> <span class="n">k</span><span class="o">),</span> <span class="n">ks</span><span class="o">))</span> <span class="n">s</span>
<span class="k">in</span>
<span class="n">exec</span> <span class="n">i</span> <span class="nc">KNil</span> <span class="n">stack</span>
</code></pre></div>
<p>Compared with <a href="#version-1.1">Version 1.1</a> and <a href="#version-1">Version 1</a>, Version 2 uses far fewer monadic combinators:</p>
<ul>
<li>Version 2 has one use of <code>return</code> (for <code>KHalt</code>, of course) and one use of <code>bind</code> (for <code>KIO</code>).</li>
<li>Versions 1 and 1.1 have four uses of <code>bind</code> and more uses of <code>return</code> than we care to count.</li>
</ul>
<p>How does this work?</p>
<ul>
<li>Pure computations (i.e. stack-related and arithmetic operations) just get pushed onto the input stack (the to-do list).</li>
<li>The impure <code>KIO</code> computation calls <code>bind</code> to glue its interpretation to the rest of the evaluation.</li>
<li>Control operators, like <code>KLoop</code>, update the control stack to dynamically determine the next instruction-sequence.</li>
</ul>
<p>Question: does this improve performance?
Answer: Yes, significantly:</p>
<div class="highlight"><pre><span></span><code>┌───────────────┬──────────┬────────────┐
│ Name │ Time/Run │ Percentage │
├───────────────┼──────────┼────────────┤
│ OCaml │ 3.62us │ 22.9% │
│ Step (V1.1) │ 15.83us │ 100.0% │
│ Step (V2) │ 6.39us │ 40.4% │
└───────────────┴──────────┴────────────┘
</code></pre></div>
<p>On this specific micro-benchmark,
Version 2 is almost three times faster than Version 1.1 and its speed is comparable (to within a factor of two) with the efficient native OCaml implementation.</p>
<p>Note that protection against stack overflows (as <a href="#version-1.1">discussed above</a>) is not required
because the new interpreter is tail recursive.
Note also that <code>step</code> Version 2 still gives us MichelBaby typechecking for free, so it is as safe (as as much static guarantee of correctness) as Version 1.</p>
<h3 id="is-that-everything"><a class="toclink" href="#is-that-everything">Is that everything?</a></h3>
<p>Granada’s new Michelson interpreter gains most of its efficiency
from the program transformation described above, but three other optimizations are also substantive:</p>
<ol>
<li>
<p>We use a local gas counter in the interpretation loop instead of the gas level found in the context.<sup id="fnref:context"><a class="footnote-ref" href="#fn:context">11</a></sup>
The OCaml compiler can represent this local gas counter in a machine register instead of a heap-allocated object; reading from and writing to a machine register is several orders of magnitude faster than for a heap-allocated object.</p>
</li>
<li>
<p>Similarly, we <em>cache</em> the top of the stack by passing it as a dedicated argument in the interpretation loop, separately from the tail of the stack.
Semantically this makes no difference — it’s still a stack! — but by this device we encourage the compiler to store the top of the stack in a machine register, potentially saving many trivial <span class="caps">RAM</span> read/writes as the top of the stack would otherwise get pushed to and then popped from memory (see <a href="(https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.5.4929&rep=rep1&type=pdf)">a paper on stack caching for interpreters</a>).</p>
</li>
<li>
<p>The Michelson interpreter includes an integrated logging tool, so that users can test smart contract code off-chain (e.g. profile their smart contracts before they go live).
This logging tool is inactive for live on-chain code, but in Florence the logger has nonzero cost even when inactive — thus, live smart contracts run slightly slower in Florence than they would if the profiler did not exist at all.
For Granada, we found a way to exploit the control stack to implement genuinely zero-cost logging.</p>
</li>
</ol>
<p><a name="tale2"></a></p>
<h2 id="tale-2-hunt-for-a-serialization-performance-bug"><a class="toclink" href="#tale-2-hunt-for-a-serialization-performance-bug">Tale 2: hunt for a serialization performance bug</a></h2>
<h3 id="doing-serialization-right-thanks-to-well-typed-encodings"><a class="toclink" href="#doing-serialization-right-thanks-to-well-typed-encodings">Doing serialization right thanks to well-typed encodings</a></h3>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Serialization"><strong>Serialization</strong></a> converts data values into a form
that can be stored or transmitted on the network (e.g. when you save a text file to disk, it gets serialized to 0s and 1s).</li>
<li><strong>Deserialization</strong> is the dual operation of reconstructing the data from its serialized representation.</li>
</ul>
<p>(De)serialization is ubiquitous in Tezos.
For example:</p>
<ul>
<li>Deserialization occurs whenever data is read from the chain (e.g. loading a script source code to execute a smart contract call).</li>
<li>Serialization occurs whenever the protocol stores information on the chain.</li>
<li>(De)serialization occurs whenever nodes communicate over the Tezos network.</li>
</ul>
<p>Nearly anything useful you might do on a blockchain involves either
reading from the chain, writing to it, or communicating with another
node — so the short list above covers pretty much everything, other
than the occasional few microseconds of pure computation.</p>
<p>The node and the economic protocol use a library named <code>data-encoding</code> — version 0.2 in Florence; version 0.3 in Granada —
that provides convenient operations to define encodings and
to automatically generate functions to convert values
into sequences of bytes or <span class="caps">JSON</span> objects,
and of course functions to decode serialized data back to values.</p>
<p>Serialization functions should not generally be written manually: it’s hard and error-prone work.
Better to generate serialization functions automatically.
To this end, <code>data-encoding</code> provides <em>combinators</em> to define a value of type <code>t encoding</code> from a value in some arbitrary serializable type <code>t</code>.
For example suppose <code>t</code> is a basic binary tree:</p>
<p><a name="tree-type"></a></p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="n">tree</span> <span class="o">=</span> <span class="nc">Leaf</span> <span class="k">of</span> <span class="kt">int</span> <span class="o">|</span> <span class="nc">Binary</span> <span class="k">of</span> <span class="n">tree</span> <span class="o">*</span> <span class="n">tree</span>
</code></pre></div>
<p>Then we can produce an encoding just by following the structure of <code>t</code> as follows:</p>
<p><a name="encoding"></a></p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="n">tree_encoding</span> <span class="o">:</span> <span class="n">tree</span> <span class="n">encoding</span> <span class="o">=</span>
<span class="n">mu</span> <span class="s2">"tree"</span> <span class="o">@@</span> <span class="c">(* `fix` is this function -> *)</span> <span class="k">fun</span> <span class="n">self</span> <span class="o">-></span>
<span class="n">union</span> <span class="o">[</span>
<span class="n">case</span>
<span class="o">(</span><span class="nc">Tag</span> <span class="mi">0</span><span class="o">)</span>
<span class="o">~</span><span class="n">title</span><span class="o">:</span><span class="s2">"Leaf"</span>
<span class="n">int31</span>
<span class="o">(</span><span class="k">function</span> <span class="nc">Leaf</span> <span class="n">x</span> <span class="o">-></span> <span class="nc">Some</span> <span class="n">x</span> <span class="o">|</span> <span class="o">_</span> <span class="o">-></span> <span class="nc">None</span><span class="o">)</span>
<span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-></span> <span class="nc">Leaf</span> <span class="n">x</span><span class="o">);</span>
<span class="n">case</span>
<span class="o">(</span><span class="nc">Tag</span> <span class="mi">1</span><span class="o">)</span>
<span class="o">~</span><span class="n">title</span><span class="o">:</span><span class="s2">"Binary"</span>
<span class="o">(</span><span class="n">obj2</span> <span class="o">(</span><span class="n">req</span> <span class="s2">"left"</span> <span class="n">self</span><span class="o">)</span> <span class="o">(</span><span class="n">req</span> <span class="s2">"right"</span> <span class="n">self</span><span class="o">))</span>
<span class="o">(</span><span class="k">function</span> <span class="nc">Binary</span> <span class="o">(</span><span class="n">left</span><span class="o">,</span> <span class="n">right</span><span class="o">)</span> <span class="o">-></span> <span class="nc">Some</span> <span class="o">(</span><span class="n">left</span><span class="o">,</span> <span class="n">right</span><span class="o">)</span> <span class="o">|</span> <span class="o">_</span> <span class="o">-></span> <span class="nc">None</span><span class="o">)</span>
<span class="o">(</span><span class="k">fun</span> <span class="o">(</span><span class="n">left</span><span class="o">,</span> <span class="n">right</span><span class="o">)</span> <span class="o">-></span> <span class="nc">Binary</span> <span class="o">(</span><span class="n">left</span><span class="o">,</span> <span class="n">right</span><span class="o">));</span>
<span class="o">]</span>
</code></pre></div>
<p>With that definition, we describe the encoding declaratively through the equation that defines the type of the value it works over:</p>
<p><a name="encoding-mu"></a></p>
<div class="math">$$
tree = \mu T. (int + T \times T)
$$</div>
<p>Combinators’ types are informative and so limit the chances of error:</p>
<p><a name="union"></a></p>
<div class="highlight"><pre><span></span><code><span class="k">val</span> <span class="n">mu</span> <span class="o">:</span> <span class="kt">string</span> <span class="o">-></span>
<span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="n">encoding</span> <span class="o">-></span> <span class="k">'</span><span class="n">a</span> <span class="n">encoding</span><span class="o">)</span> <span class="o">-></span>
<span class="k">'</span><span class="n">a</span> <span class="n">encoding</span>
<span class="k">val</span> <span class="n">case</span> <span class="o">:</span> <span class="n">title</span><span class="o">:</span><span class="kt">string</span> <span class="o">-></span> <span class="n">case_tag</span> <span class="o">-></span>
<span class="k">'</span><span class="n">a</span> <span class="n">encoding</span> <span class="o">-></span> <span class="o">(</span><span class="k">'</span><span class="n">t</span> <span class="o">-></span> <span class="k">'</span><span class="n">a</span> <span class="n">option</span><span class="o">)</span> <span class="o">-></span> <span class="o">(</span><span class="k">'</span><span class="n">a</span> <span class="o">-></span> <span class="k">'</span><span class="n">t</span><span class="o">)</span> <span class="o">-></span>
<span class="k">'</span><span class="n">t</span> <span class="n">case</span>
<span class="k">val</span> <span class="n">union</span> <span class="o">:</span> <span class="o">?</span><span class="n">tag_size</span><span class="o">:[`</span><span class="nc">Uint8</span> <span class="o">|</span> <span class="o">`</span><span class="nc">Uint16</span><span class="o">]</span> <span class="o">-></span>
<span class="k">'</span><span class="n">t</span> <span class="n">case</span> <span class="kt">list</span> <span class="o">-></span>
<span class="k">'</span><span class="n">t</span> <span class="n">encoding</span>
</code></pre></div>
<p>So <code>data-encoding</code> makes serialization right:
we avoid writing error-prone boilerplate and
thanks to combinators</p>
<ul>
<li>the input required from the programmer is minimal (thanks to their expressivity),</li>
<li>the results are fairly likely to be correct (thanks to the combinators’ precise types), and</li>
<li>the results are safe (again, thanks to static typing).</li>
</ul>
<h3 id="how-costly-is-this-abstraction"><a class="toclink" href="#how-costly-is-this-abstraction">How costly is this abstraction?</a></h3>
<p>Serialization with <code>data-encoding</code> version 0.2 (as in Florence, and thus before Granada) may have been “right”, but it was not particularly fast.
Typically, if we compare the time required
to encode a binary tree with <span class="math">\(2^{20}\)</span> nodes using <code>data-encoding</code>
with the time taken to encode the same tree using <a href="https://ocaml.org/api/Marshal.html"><code>Marshal</code>
(the unsafe serialization module of OCaml standard library)</a><sup id="fnref:marshal"><a class="footnote-ref" href="#fn:marshal">12</a></sup>,
we get the following numbers:</p>
<p><a name="benchmark-data-encoding-1"></a></p>
<div class="highlight"><pre><span></span><code>┌───────────────────┬──────────┬────────────┐
│ Name │ Time/Run │ Percentage │
├───────────────────┼──────────┼────────────┤
│ Marshal │ 67.46ms │ 30.4% │
│ data-encoding 0.2 │ 221.82ms │ 100.0% │
└───────────────────┴──────────┴────────────┘
</code></pre></div>
<p><code>data-encoding</code> version 0.2 is roughly three times slower that <code>Marshal</code> in that example.<sup id="fnref:test-2"><a class="footnote-ref" href="#fn:test-2">13</a></sup></p>
<p>One might assume this is just the price of having
a nice abstraction for and a well-typed implementation of serialization.
After all, <code>Marshal</code> is untyped and implemented in C, and low-level programming in C can be fast.</p>
<p>Surprisingly, this is not the case.
Using <a href="https://web.archive.org/web/20210618120941/https://www.brendangregg.com/perf.html">the linux <code>perf</code> command</a> we can observe that a specific function in <code>data-encoding</code> takes half of the execution time during writing: <a href="https://archive.softwareheritage.org/browse/origin/content/?branch=refs/tags/0.2&origin_url=https://gitlab.com/nomadic-labs/data-encoding/&path=src/encoding.ml#L752"><code>check_cases</code></a>.<sup id="fnref:permalink"><a class="footnote-ref" href="#fn:permalink">14</a></sup></p>
<p><code>check_cases</code> checks that the <a href="#union"><code>data-encoding</code><span class="quo">‘</span>s combinator for <code>union</code> types</a>
is properly applied to a <a href="https://archive.softwareheritage.org/browse/origin/content/?branch=refs/tags/0.2&origin_url=https://gitlab.com/nomadic-labs/data-encoding/&path=src/encoding.ml#L752">nonempty</a> list of <a href="https://archive.softwareheritage.org/browse/origin/content/?branch=refs/tags/0.2&origin_url=https://gitlab.com/nomadic-labs/data-encoding/&path=src/encoding.ml#L752">disjoint</a> cases.</p>
<p>It’s surprising to see <code>check_cases</code> called during serialization of concrete values, because one would have expected it to be called just once when the definition of the encoding is processed.</p>
<p>In fact, things are even worse than they appear because <code>check_cases</code> has quadratic complexity with respect to the number of cases.
Thus execution time degrades further if we add a new data constructors to our type for binary trees:</p>
<p><a name="benchmark-data-encoding-2"></a></p>
<div class="highlight"><pre><span></span><code>┌───────────────────┬──────────┬───────────┐
│ Name │ Time/Run │Percentage │
├───────────────────┼──────────┼───────────┤
│ Marshal │ 74.25ms │ 25.4% │
│ data-encoding 0.2 │ 292.16ms │ 100.0% │
└───────────────────┴──────────┴───────────┘
</code></pre></div>
<p>For datatypes with five or more constructors, the performance of <code>data-encoding</code> seems to bottom out at roughly ten times slower than <code>Marshal</code>.</p>
<p>So, why is <code>check_cases</code> called during serialization of concrete values? The investigation is simple since the definition of <code>tree encoding</code> uses only two combinators: <code>union</code> and <code>mu</code>.</p>
<p><code>check_cases</code> is called during serialization of concrete values because of a bug in <code>mu</code> which we will explain now.</p>
<p>Fortunately, <a href="https://archive.softwareheritage.org/browse/origin/content/?branch=refs/tags/0.2&origin_url=https://gitlab.com/nomadic-labs/data-encoding/&path=src/binary_writer.ml#L330">the case for <code>mu</code>-based encodings in the function <code>write_rec</code></a> is a mere one-liner:</p>
<p><a name="writemu"></a></p>
<div class="highlight"><pre><span></span><code><span class="o">|</span> <span class="nc">Mu</span> <span class="o">{</span><span class="n">fix</span><span class="o">;</span> <span class="o">_}</span> <span class="o">-></span>
<span class="n">write_rec</span> <span class="o">(</span><span class="n">fix</span> <span class="n">e</span><span class="o">)</span> <span class="n">state</span> <span class="k">value</span>
</code></pre></div>
<p>This line says that <code>fix</code> (the function passed to <code>mu</code>) is called to get the encoding of the value to be encoded.
Here <code>Mu</code> is the internal data constructor used in the function <code>mu</code>.</p>
<p>Let us see what this <code>fix</code> function is when <code>write_rec</code> is executed on our example for <a href="#encoding"><code>tree encoding</code></a>:</p>
<div class="highlight"><pre><span></span><code> <span class="mi">1</span><span class="o">.</span> <span class="k">let</span> <span class="n">tree_encoding</span> <span class="o">:</span> <span class="n">tree</span> <span class="n">encoding</span> <span class="o">=</span>
<span class="mi">2</span><span class="o">.</span> <span class="n">mu</span> <span class="s2">"tree"</span> <span class="o">@@</span> <span class="c">(* `fix` is this function -> *)</span> <span class="k">fun</span> <span class="n">self</span> <span class="o">-></span>
<span class="mi">3</span><span class="o">.</span> <span class="n">union</span> <span class="o">[</span>
<span class="mi">4</span><span class="o">.</span> <span class="n">case</span>
<span class="mi">5</span><span class="o">.</span> <span class="o">(</span><span class="nc">Tag</span> <span class="mi">0</span><span class="o">)</span>
<span class="mi">6</span><span class="o">.</span> <span class="o">~</span><span class="n">title</span><span class="o">:</span><span class="s2">"Leaf"</span>
<span class="mi">7</span><span class="o">.</span> <span class="n">int31</span>
<span class="mi">8</span><span class="o">.</span> <span class="o">(</span><span class="k">function</span> <span class="nc">Leaf</span> <span class="n">x</span> <span class="o">-></span> <span class="nc">Some</span> <span class="n">x</span> <span class="o">|</span> <span class="o">_</span> <span class="o">-></span> <span class="nc">None</span><span class="o">)</span>
<span class="mi">9</span><span class="o">.</span> <span class="o">(</span><span class="k">fun</span> <span class="n">x</span> <span class="o">-></span> <span class="nc">Leaf</span> <span class="n">x</span><span class="o">);</span>
<span class="mi">10</span><span class="o">.</span> <span class="n">case</span>
<span class="mi">11</span><span class="o">.</span> <span class="o">(</span><span class="nc">Tag</span> <span class="mi">1</span><span class="o">)</span>
<span class="mi">12</span><span class="o">.</span> <span class="o">~</span><span class="n">title</span><span class="o">:</span><span class="s2">"Binary"</span>
<span class="mi">13</span><span class="o">.</span> <span class="o">(</span><span class="n">obj2</span> <span class="o">(</span><span class="n">req</span> <span class="s2">"left"</span> <span class="n">self</span><span class="o">)</span> <span class="o">(</span><span class="n">req</span> <span class="s2">"right"</span> <span class="n">self</span><span class="o">))</span>
<span class="mi">14</span><span class="o">.</span> <span class="o">(</span><span class="k">function</span> <span class="nc">Binary</span> <span class="o">(</span><span class="n">left</span><span class="o">,</span> <span class="n">right</span><span class="o">)</span> <span class="o">-></span> <span class="nc">Some</span> <span class="o">(</span><span class="n">left</span><span class="o">,</span> <span class="n">right</span><span class="o">)</span> <span class="o">|</span> <span class="o">_</span> <span class="o">-></span> <span class="nc">None</span><span class="o">)</span>
<span class="mi">15</span><span class="o">.</span> <span class="o">(</span><span class="k">fun</span> <span class="o">(</span><span class="n">left</span><span class="o">,</span> <span class="n">right</span><span class="o">)</span> <span class="o">-></span> <span class="nc">Binary</span> <span class="o">(</span><span class="n">left</span><span class="o">,</span> <span class="n">right</span><span class="o">));</span>
<span class="mi">16</span><span class="o">.</span> <span class="o">]</span>
</code></pre></div>
<p>This is a correct encoding function!
In fact the return type of the <code>fix</code> function is <code>tree encoding</code>
so it makes sense to recursively call <code>write_rec</code> to encode <code>value</code> in the interpretation of <a href="#writemu"><code>mu</code></a>
since it is of <a href="#tree-type">type <code>tree</code></a>.
Besides, <code>self</code> will be equal to <code>e</code> in the interpretation of <a href="#writemu"><code>mu</code></a>,
that is <code>Mu {fix; ...}</code>
which is also consistent because <code>tree_encoding</code> is used to serialize the sub-trees.</p>
<p>However, even though the definition of <code>tree_encoding</code> is functionally correct,
it has a hidden performance bug.
Each time we write a tree,
we execute <code>union</code> (see line 3 in the code above), and <a href="https://gitlab.com/nomadic-labs/data-encoding/-/blob/875b189e841bcb29e6e54d572d0ae8e829e9ea15/src/encoding.ml#L773"><code>union</code></a> carries out some sanity checks, including the aforementioned <code>check_cases</code>.
The encoding for trees is an immutable value, so its definition need only be checked once — not each time a tree is serialized!</p>
<h3 id="how-to-fix-fix"><a class="toclink" href="#how-to-fix-fix">How to fix <code>fix</code>?</a></h3>
<p>Now that we understand the problem with <code>fix</code>, how can we address it?</p>
<p>The idea is simple: the construction of the encoding for an arbitrary recursive type (of which the binary trees considered above are an example) should execute <code>fix</code> only once
and then produce an encoding that we can trust to serialize an arbitrary number of data elements of that type,
with no further checks.</p>
<p>This works because an encoding is an immutable value
so executing <code>fix</code> on <code>e</code> always returns the same value.
On the implementation side,
we just need to call <code>fix e</code> once,
then remember the result in a local reference
and return the content of this reference in subsequent evaluations of <code>fix e</code>.
Programmers may recognize this as the common technique called <a href="https://en.wikipedia.org/wiki/Memoization"><strong>memoization</strong> (caching the results of function calls)</a>.</p>
<p>Here is an excerpt of the corrected <code>mu</code> that illustrates how we proceed:</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="n">self</span> <span class="o">=</span> <span class="n">ref</span> <span class="nc">None</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">fix_f</span> <span class="o">=</span> <span class="n">fix</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">fix</span> <span class="n">e</span> <span class="o">=</span>
<span class="k">match</span> <span class="o">!</span><span class="n">self</span> <span class="k">with</span>
<span class="o">|</span> <span class="nc">Some</span> <span class="o">(</span><span class="n">e0</span><span class="o">,</span> <span class="n">e'</span><span class="o">)</span> <span class="k">when</span> <span class="n">e</span> <span class="o">==</span> <span class="n">e0</span> <span class="o">-></span>
<span class="n">e'</span>
<span class="o">|</span> <span class="o">_</span> <span class="o">-></span>
<span class="k">let</span> <span class="n">e'</span> <span class="o">=</span> <span class="n">fix_f</span> <span class="n">e</span> <span class="k">in</span>
<span class="n">self</span> <span class="o">:=</span> <span class="nc">Some</span> <span class="o">(</span><span class="n">e</span><span class="o">,</span> <span class="n">e'</span><span class="o">)</span> <span class="o">;</span>
<span class="n">e'</span>
<span class="k">in</span>
<span class="o">...</span>
</code></pre></div>
<p><code>self</code> remembers that the <code>fix e = e'</code>.
This allows us to return <code>e'</code>
when <code>fix</code> is called with <code>e</code> again
instead of recomputing <code>fix e</code>.</p>
<p>We re-run <a href="#benchmark-data-encoding-1">our first benchmark</a> on <code>data-encoding</code> version 0.2 but with a correct mu and see that this patch significantly improvemes the performance of <code>data-encoding</code>:</p>
<div class="highlight"><pre><span></span><code>┌────────────────────────────────┬──────────┬────────────┐
│ Name │ Time/Run │ Percentage │
├────────────────────────────────┼──────────┼────────────┤
│ Marshal │ 69.19ms │ 61.5% │
│ data-encoding with correct mu │ 112.42ms │ 100.0% │
└────────────────────────────────┴──────────┴────────────┘
</code></pre></div>
<p>We reduced the execution time, and also our encoding is now less sensitive to the number of cases in the encoding.
Indeed, the <a href="#benchmark-data-encoding-2">second benchmark</a> (in which our type for binary trees is extended with additional data constructors) performs almost as well (112.42ms vs 128.18ms):</p>
<div class="highlight"><pre><span></span><code>┌───────────────────────────────┬──────────┬────────────┐
│ Name │ Time/Run │ Percentage │
├───────────────────────────────┼──────────┼────────────┤
│ Marshal │ 74.14ms │ 57.8% │
│ data-encoding with correct mu │ 128.18ms | 100.0% │
└───────────────────────────────┴──────────┴────────────┘
</code></pre></div>
<h3 id="is-that-everything_1"><a class="toclink" href="#is-that-everything_1">Is that everything?</a></h3>
<p>The optimizations described above account for the bulk of the performance improvements moving from <code>data-encoding 0.2</code> in Florence to <code>data-encoding 0.3</code> in Granada.</p>
<p><code>data-encoding 0.3</code> does include other incremental optimizations, which yield real but less significant speedups, and we can sum it up in a little equation as follows: </p>
<div class="highlight"><pre><span></span><code>data-encoding 0.3 in Granada = data-encoding 0.2 from Florence +
correct mu +
some other optimizations
</code></pre></div>
<p>Yet marginal gains are still gains and <a href="https://web.archive.org/web/20210612181818/https://jamesclear.com/marginal-gains">can still accumulate</a>, and putting all of them together we were able to reach nearly the same level of efficiency as <code>Marshal</code> for important encodings — in particular the one dedicated to the <code>Micheline</code> format, a central data representation in the Tezos protocol.</p>
<h2 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h2>
<p>We said at the start of this blog post that we followed a process of <em>“Make it work, make it right, make it fast”</em>.</p>
<p>These three qualities, far from being in opposition, complemented one another to make our optimizations possible and safe.
We <em>needed</em> the codebase to be of high quality — having a sound architecture and using appropriate abstractions — to find, perform, and check our optimizations.
Clean, well-engineered code meant that we could understand the code, effectively profile it, locate sources of inefficiency, and then apply <em>local</em> rewrites to improve efficiency.</p>
<p>Code that is right, is easier to optimize.
Even in the context of a critical system like Tezos, where correctness and security have the highest priority — <em>especially</em> the context of a critical system — making it right also helps to make it fast.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:economic-protocol">
<p>Some background: the Tezos blockchain has what amounts to an operating system layer called the <strong>economic protocol</strong>, which is unusual amongst blockchains in that it is upgradable by stake-weighted community vote.</p>
<p>The currently-live economic protocol is <a href="https://tezos.gitlab.io/protocols/009_florence.html">Florence</a>, and an upgrade to <a href="https://tezos.gitlab.io/protocols/010_granada.html">Granada</a> is now scheduled for August 2021.
So this blog post can be thought of as an inside peek into how a significant performance optimization was attained for Tezos’ next big <span class="caps">OS</span> upgrade.
Developing such updates is a large part of Nomadic Labs’ day-to-day activities. <a class="footnote-backref" href="#fnref:economic-protocol" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:gas">
<p>A blockchain is “just” a distributed state machine and smart contracts are “just” programs that run on that machine to modify the state.
“Gas” is used to bound computation and so prevent denial of service attacks: a smart contract receives a gas budget when started, and if it exhausts its gas, it gets terminated.
Lower gas consumption is a corollary of optimizing on-chain computation to make it run more efficiently.
Or to put it another way: optimizing Tezos to run faster on a given piece of actual hardware, means that more useful on-chain computation can be fit into a given gas budget. <a class="footnote-backref" href="#fnref:gas" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:test-1">
<p>Test suite here: <a href="https://research-development.nomadic-labs.com/files/202106-granada-gas-improvement/bench.sh.html">bench.sh</a>, <a href="https://research-development.nomadic-labs.com/files/202106-granada-gas-improvement/dexter.tz.html">dexter.tz</a>, <a href="https://research-development.nomadic-labs.com/files/202106-granada-gas-improvement/fa1.2.tz.html">fa1.2.tz</a>, <a class="footnote-backref" href="#fnref:test-1" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:bytecode">
<p>You can use another language to write your smart contract, but to <em>execute</em> it on the Tezos blockchain you’ll need to compile to Michelson. So does that make Michelson a bytecode language? Yes, but just calling it that somewhat belies its power: we describe it as a “strongly-typed low-level language” instead, which arguably captures the spirit of things somewhat better. <a class="footnote-backref" href="#fnref:bytecode" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:simplification">
<p>The subtle simplification here is that <code>monad</code> does not quite represent a single state; it represents a <em>state-change</em> (i.e. a side-effect). However, if we assume that all computations start from an initial empty state, then a state change can be identified with the result of applying the state change to an initial empty state. <a class="footnote-backref" href="#fnref:simplification" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:scribble">
<p>That’s like running a calculation on a calculator, reading off the result … then switching off the calculator. The calculator had a state but we don’t care about it any more because we have the result we wanted. <a class="footnote-backref" href="#fnref:scribble" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:bytes">
<p>We’d like to call them <em>bytes of state</em> … but that would be confusing. Perhaps <em>bits of state</em>? <a class="footnote-backref" href="#fnref:bytes" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn:object">
<p><em>Object-level language</em> here just means that MichelBaby is the language we are implementing. Contrast this with the <em>meta-level language</em> which is the language <em>in which</em> we are writing the implementation; OCaml, in this case. <a class="footnote-backref" href="#fnref:object" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
<li id="fn:python">
<p>Python does this for you whether you want it to or not.
There’s a <a href="https://web.archive.org/web/20210503063240/https://www.geeksforgeeks.org/python-handling-recursion-limit/">system-wide (mutable) limit of 1000 on recursive calls</a>.
The implication is that if your function has recursed that many times, it’s actually doing a loop (i.e. an iteration).
Tail recursion is a functional programmer’s version of iteration and in particular, OCaml automatically recognizes and optimizes tail recursive calls to avoid stack allocation, so that a tail recursive OCaml program is operationally just a (generalisation of an) iterative loop. <a class="footnote-backref" href="#fnref:python" title="Jump back to footnote 9 in the text">↩</a></p>
</li>
<li id="fn:limit">
<p><span class="dquo">“</span>Good” here means something specific:
In practice, smart contracts are subject to gas limits — specifically, the current <em>Operation Gas Limit</em> for the currently-live Florence protocol is <a href="https://tzstats.com/protocols/PsFLorenaUUuikDWvMDr6fGBRG8kt3e3D3fHoXK1j1BFRxeSH4i">1,040,000</a>; see also this limit <a href="https://archive.softwareheritage.org/browse/origin/content/?origin_url=https://gitlab.com/tezos/tezos&path=src/proto_009_PsFLoren/lib_parameters/default_parameters.ml#L38">directly in the relevant line in the Florence source code</a> — so a smart contract that gobbles our interpreter’s resources is likely to be terminated for exhausting its gas allocation <em>anyway</em> by the ambient Tezos abstract machine, and this would normally happen before the local hardware that is running the Tezos node on which our abstract machine happens to be executing, runs out of stack space.
Thus a “good” limit is </p>
<ul>
<li><em>large enough</em> to allow complete gas exhaustion for practical smart contracts that run on-chain (so in practice it would never actually be reached), yet </li>
<li><em>small enough</em> to prevent stack overflows on the underlying execution environment (so our code has predictable, controllable behaviour that we can reason about).</li>
</ul>
<p>Does the fact that we do not expect this branch to be reached in practice make it irrelevant?
Not really: for <em>impractical</em> smart contracts, stack could still be exhausted before gas. For instance, a deep recursion of sufficiently cheap operations (e.g. <code>PUSH 1; DROP</code>) might still overflow the call stack before gas is exhausted.
So we do still have to detect and fail deep recursions — just in case. <a class="footnote-backref" href="#fnref:limit" title="Jump back to footnote 10 in the text">↩</a></p>
</li>
<li id="fn:context">
<p>The context is the state of the blockchain used to validate a block. <a class="footnote-backref" href="#fnref:context" title="Jump back to footnote 11 in the text">↩</a></p>
</li>
<li id="fn:marshal">
<p>The Marshal module of OCaml is untyped. For this reason, the module must trust the programmer that
some serialized bytes can be turned into a value of a given type. If the programmer is wrong, the program may
crash. <a class="footnote-backref" href="#fnref:marshal" title="Jump back to footnote 12 in the text">↩</a></p>
</li>
<li id="fn:test-2">
<p>Test file here: <a href="https://research-development.nomadic-labs.com/files/202106-granada-gas-improvement/comparing-marshal-and-data-encoding-0.2.ml.html">comparing-marshal-and-data-encoding-0.2.ml.html</a>. <a class="footnote-backref" href="#fnref:test-2" title="Jump back to footnote 13 in the text">↩</a></p>
</li>
<li id="fn:permalink">
<p>The files linked are at <a href="https://gitlab.com/nomadic-labs/data-encoding/-/tree/0.2/src">https://gitlab.com/nomadic-labs/data-encoding/-/tree/0.2/src</a> at time of writing. However, we will give permalinks to an archived snapshot of the repository at <a href="https://archive.softwareheritage.org/browse/origin/content/?branch=refs/tags/0.2&origin_url=https://gitlab.com/nomadic-labs/data-encoding/&path=src/">https://archive.softwareheritage.org/browse/origin/content/?branch=refs/tags/0.2&origin_url=https://gitlab.com/nomadic-labs/data-encoding/&path=src/</a>. <a class="footnote-backref" href="#fnref:permalink" title="Jump back to footnote 14 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Meanwhile at Nomadic Labs #122021-07-23T10:00:00+02:002021-07-23T10:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-07-23:/meanwhile-at-nomadic-labs-12.html<p>A summary of Nomadic Labs activities in Q2 2021</p><p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>Welcome to our <a href="https://research-development.nomadic-labs.com/blog.html?category=meanwhile">meanwhile series</a>, the ongoing story of Nomadic Labs’ amazing adventures in the Tezos blockchain space.
This post is a recap of our activities in the second quarter of 2021, following on from our <a href="https://research-development.nomadic-labs.com/a-review-of-nomadic-labs-in-2020.html">2020 recap</a> and our <a href="https://research-development.nomadic-labs.com/meanwhile-at-nomadic-labs-11.html">2021 first-quarter Meanwhile</a>.
As always, you can find out more about us here:</p>
<blockquote>
<p><a href="https://twitter.com/labosnomades?lang=en">Twitter @LabosNomades</a> ~
<a href="https://www.nomadic-labs.com">Website</a> ~
<a href="https://fr.linkedin.com/company/nomadic-labs">LinkedIn</a> ~
<a href="https://research-development.nomadic-labs.com/blog.html">Technical blog</a> ~
<a href="https://gitlab.com/nomadic-labs">GitLab repo</a> </p>
</blockquote>
<p>So here’s what we’ve been up to these past three months:</p>
<ul>
<li><a href="#octez">Octez</a></li>
<li><a href="#adoption-and-support">Adoption and Support</a></li>
<li><a href="#training">Training</a></li>
<li><a href="#umami">Umami</a></li>
<li><a href="#announcing-a-new-logo-and-website">Announcing: a new logo and website</a></li>
<li><a href="#culture-and-growth">Culture and growth</a></li>
<li><a href="#announcing-phd-student-intern-and-apprentice-interviews">Announcing: PhD student, Intern, and apprentice interviews</a></li>
<li><a href="#protocol-upgrade-florence-activated-granada-proposed">Protocol upgrade: Florence activated, Granada proposed</a></li>
<li><a href="#the-florence-upgrade">The Florence upgrade</a></li>
<li><a href="#the-granada-upgrade-proposal">The Granada upgrade proposal</a></li>
<li><a href="#tenderbakenet">Tenderbakenet</a></li>
<li><a href="#testing">Testing</a></li>
<li><a href="#interviews-and-papers">Interviews and Papers</a></li>
<li><a href="#nl-research-seminars"><span class="caps">NL</span> research seminars</a></li>
<li><a href="#tezos-is-3">Tezos is 3</a></li>
<li><a href="#à-la-prochaine">À la prochaine</a></li>
</ul>
<h2 id="octez"><a class="toclink" href="#octez">Octez</a></h2>
<p>We are pleased to announce <a href="https://research-development.nomadic-labs.com/announcing-octez.html">Octez</a>.
Octez is a new name for a veteran implementation of Tezos which had previously been known just by its version number and by <a href="https://gitlab.com/tezos/tezos/">a GitLab repo</a>.</p>
<p>At time of writing, the latest version of Octez is <a href="https://tezos.gitlab.io/releases/version-9.html">Octez Version 9.4</a>,
and the latest release candidate is <a href="https://tezos.gitlab.io/releases/version-10.html">Octez Version 10.0~rc1</a>.</p>
<p><a href="https://gitlab.com/tezos/tezos/">Here is the Octez GitLab repo</a>.
Feel free to <a href="https://tezos.gitlab.io/introduction/howtoget.html">get the Octez Tezos implementation</a> and join the Tezos blockchain!</p>
<h2 id="adoption-and-support"><a class="toclink" href="#adoption-and-support">Adoption and Support</a></h2>
<p>Our adoption and support teams<sup id="fnref:two"><a class="footnote-ref" href="#fn:two">1</a></sup> have been hard at work developing relationships, and thanks to their dilligence we are proud to announce that:</p>
<ul>
<li>On 15 April 2021 <a href="https://www.societegenerale.com/en/news/press-release/first-structured-product-public-blockchain">Forge Capital Markets (a subsidiary of Société Générale) created a structured product on Tezos public blockchain</a>. This five million Euro pilot scheme is designed in particular to demonstrate the legal, regulatory, and operational feasibility of issuing complex financial instruments (structured products) on a public blockchain. </li>
<li>On 20 April 2021 <a href="https://www.nomadic-labs.com/news/ubisoft-is-joining-the-tezos-ecosystem-as-a-corporate-baker/">Ubisoft (a leading computer gaming company) became an institutional baker</a> via its Strategic Innovation Lab, whose mission is to <a href="https://news.ubisoft.com/en-us/article/3niQ2Mq4koWNmZNZcdZ0TB/how-the-strategic-innovation-lab-helps-ubisoft-embrace-its-future">help prepare Ubisoft for the future by exploring innovative technologies</a>. As a Tezos corporate baker, Ubisoft will experiment with the liquid proof-of-stake consensus algorithm and research the potential of blockchain technology for the future of gaming.<sup id="fnref:baker"><a class="footnote-ref" href="#fn:baker">2</a></sup></li>
<li>Nomadic Labs has been helping <a href="https://connect.arteia.com">Arteïa</a> to sell artworks as non-fungible tokens on the <a href="https://www.hicetnunc.xyz/about">hic et nunc</a> platform (<a href="https://adoption-support.nomadic-labs.com/case-studies/arteia/">case study here</a>). On May 8, <a href="https://twitter.com/ArteiaSolutions/status/1390588226100989953?s=20">a number of artworks by Benjamin Spark were “dropped” on the platform</a>. </li>
<li>On 10 May, <a href="https://adan.eu/en/general-assembly-faustine-fleuret-elected-presiden">Nomadic Labs was re-elected as a board member of the association <span class="caps">ADAN</span></a> (an <a href="https://adan.eu/en/">industry federation in the crypto-asset and blockchain sector in France and across Europe</a>).</li>
<li>On 3 June 2021 <a href="https://www.nomadic-labs.com/news/nomadic-labs-x-walchain/">Nomadic Labs joined WalChain</a>, a Wallonian business network of blockchain startups, investment funds, clusters, research centres, and Wallonian universities. As Oana Ladret Piciorus, Managing Director of Nomadic Labs said: <em>“We are happy to join WalChain and hope to bring to this ecosystem our extensive knowledge of the Tezos protocol and of course offer our support to companies in this region looking to explore it.”</em> </li>
<li>In June 2021 the association <em>Pour Que Marseille Vive!</em> and <em>Equisafe</em> deployed the first non-fungible token (<span class="caps">NFT</span>) of a physical work of art (by the Marseille artist Deniz Doruk) on the Tezos public blockchain. You can view it <a href="https://xtz.news/digital-securities-news/equisafe-tokenized-its-first-piece-of-physical-art-on-tezos/">here</a>. Watch this space! </li>
<li>MacLaren is working on Tezos NFTs with our adoption team. See this <a href="https://www.mclaren.com/racing/inside-the-mtc/nft-guide/"><span class="caps">NFT</span>, it’s as easy as 1-2-3</a> howto (dated 17 June).</li>
<li>We have an ongoing collaboration with <a href="https://www.blockstart.eu">BlockStart</a>, the Blockchain Partnership Programme, and we are pleased to announce that on June 17, Thibault Chessé, Alexia Mertinel, and Hadrien Zerah became <a href="https://www.blockstart.eu/mentors/">blockchain mentors with Blockstart</a>, helping to mentor blockchain start-ups during the Prototype and Pilot stages.</li>
</ul>
<p><a name="train"></a></p>
<h2 id="training"><a class="toclink" href="#training">Training</a></h2>
<p>Training is key to uptake and adoption and in May/June 2021 Nomadic Labs, in collaboration with colleagues from the <a href="https://smartpy.io">SmartPy</a>, <a href="https://ligolang.org">Ligo</a> and <a href="https://archetype-lang.org">Archetype</a> language teams,
organised two online training sessions on Smart Contract programming (in French: “Développer des Smart Contrats sur Tezos”) for a total of just over thirty developers from various French companies interested in doing projects based on the Tezos blockchain.</p>
<p>The courses were well-received and
we plan to continue to run training courses in both French and English.
If you’re interested, please just send us an e-mail at <a href="mailto:training@nomadic-labs.com">training@nomadic-labs.com</a>.</p>
<h2 id="umami"><a class="toclink" href="#umami">Umami</a></h2>
<p>The Umami Team at Nomadic Labs released the Beta version of the <a href="https://umamiwallet.com">Umami wallet</a> in April 2021.
This Tezos <a href="https://en.wikipedia.org/wiki/Cryptocurrency_wallet">cryptocurrency wallet</a> is designed as a power tool, built <a href="https://medium.com/umamiwallet/introducing-the-umami-wallet-4c5d4201ce5">by OCaml developers for OCaml developers</a> using <a href="https://en.wikipedia.org/wiki/Reason_(programming_language)">ReasonML</a>, and supporting all the native features of the Tezos protocol, including multiple accounts, tokens, batch transactions, and delegation — with <a href="https://umamiwallet.com/#features">more features</a> in the pipeline.</p>
<p>You can <a href="https://umamiwallet.com/#download">download Umami here</a>.
For more information see:</p>
<ul>
<li><a href="https://medium.com/umamiwallet">A Umami Wallet page on Medium</a>.</li>
<li>A <a href="https://medium.com/umamiwallet/introducing-the-umami-wallet-4c5d4201ce5">short essay on the purpose of Umami</a>. </li>
<li>The <a href="https://gitlab.com/nomadic-labs/umami-wallet/umami">Umami GitLab repo</a>.</li>
<li>The list of <a href="https://gitlab.com/nomadic-labs/umami-wallet/umami/-/releases">Umami releases</a>. </li>
</ul>
<h2 id="announcing-a-new-logo-and-website"><a class="toclink" href="#announcing-a-new-logo-and-website">Announcing: a new logo and website</a></h2>
<p>Nomadic Labs is happy to introduce to the world its <a href="https://twitter.com/LabosNomades/status/1366750819908001795">new logo</a> and its new <a href="https://twitter.com/LabosNomades/status/1408061339440861195">modern and accessible website</a>.</p>
<p>Since our team <a href="https://infrachain.com/nomadic-labs/">formed in 2018</a>, the Tezos project has grown considerably, and our own identity has grown along with it.
While we remain primarily an OCaml programming house, we have seen our activities diversify with the addition of teams dedicated to adoption, support, and training.
Our new image encapsulates our dedication to technical excellence, community and commitment, which are the core values for our activities.</p>
<table style="width:100%">
<tr>
<th>
</th>
<th>Our 2018 look</th>
<th>Our new 2021 look</th>
</tr>
<tr>
<td style="width:10%">
</td>
<td><img src="images/meanwhile_202107/nomadic-logo-old.png" alt="2018 logo" style="width:80%"></td>
<td><img src="images/meanwhile_202107/nomadic-logo-new.png" alt="New logo" style="width:80%"></td>
</tr>
</table>
<h2 id="culture-and-growth"><a class="toclink" href="#culture-and-growth">Culture and growth</a></h2>
<p>Since April 2021 we are delighted to have been joined by five new hires and four interns (see next item), bringing our count of full-time employees to 67. </p>
<h2 id="announcing-phd-student-intern-and-apprentice-interviews"><a class="toclink" href="#announcing-phd-student-intern-and-apprentice-interviews">Announcing: PhD student, Intern, and apprentice interviews</a></h2>
<p>It’s the people in a workplace that make it a good place for people to work in.</p>
<p>Thus we are extremely pleased at Nomadic Labs to host interns (<em>stagiaires</em>) and apprentices (<em>apprentis</em>), and to supervise some PhD students in collaboration with the local universities in Paris: our contribution to educating the next generation and to getting fresh perspectives on our own work.</p>
<p>Our interns, apprentices, and PhD students come from many backgrounds, and each has their own unique story to tell, so we’ve introduced a <a href="https://research-development.nomadic-labs.com/blog.html?category=people">‘people’ category</a> to the Nomadic Labs blog to host interviews with our valued guests.
Interviews so far have included:</p>
<ul>
<li><a href="https://research-development.nomadic-labs.com/three-questions-to-nomadic-labs-interns-corentin-calmels.html">Corentin Calmels, an intern working with the adoption team on new business solutions</a>.</li>
<li><a href="https://research-development.nomadic-labs.com/five-questions-to-nomadic-labs-phds-paul-laforgue.html">Paul Laforgue, a PhD student working with the Shell team on specification and verification of message-passing distributed systems using choreographies</a>.</li>
<li><a href="https://research-development.nomadic-labs.com/five-questions-to-nomadic-labs-phds-guillaume-bau.html">Guillaume Bau, a PhD student working on abstract interpretation for Michelson smart contracts</a>.</li>
<li><a href="https://research-development.nomadic-labs.com/five-questions-to-nomadic-labs-phds-colin-gonzalez.html">Colin Gonzalez, a PhD student working on spreadsheet-like paradigms for practical smart contract programming languages</a>.</li>
</ul>
<h2 id="protocol-upgrade-florence-activated-granada-proposed"><a class="toclink" href="#protocol-upgrade-florence-activated-granada-proposed">Protocol upgrade: Florence activated, Granada proposed</a></h2>
<p>On <a href="[11 May 2021](https://tzstats.com/election/27)">11 May 2021 Florence was activated</a>, and on <a href="https://tzstats.com/election/29">25 May 2021 Granada was proposed</a>.</p>
<p>The Tezos <em>protocol environment</em> enjoys <a href="https://forum.tezosagora.org/t/regular-scheduling-for-our-tezos-proposals/2331">regular performance and security upgrades</a>.<sup id="fnref:library"><a class="footnote-ref" href="#fn:library">3</a></sup>
How this happens concretely is that a <a href="https://tezos.com/docs/learn/what-is-tezos#self-amendment">self-amendment mechanism</a> is activated to propose an upgrade the protocol — and because Tezos is an open community, protocol upgrades are approved by community vote.
This means that upgrades can only happen when you, the Tezos community, vote that it be so; which is why you’ll notice we only ever talk about us making upgrade <em>proposals</em>.</p>
<p>Most recently, </p>
<ul>
<li><a href="https://research-development.nomadic-labs.com/delphi-official-release.html">Delphi</a> was activated on <a href="https://tzstats.com/election/24">12 November 2020</a> (<a href="https://tzstats.com/1212417">block height 1,212,417</a>; <a href="https://tzstats.com/cycle/296">cycle 296</a>; <a href="https://research-development.nomadic-labs.com/delphi-changelog.html">changelog</a>; <a href="https://research-development.nomadic-labs.com/smarter-contracts-thanks-to-delphi-part-12.html">significance of the upgrade</a>), and</li>
<li><a href="https://forum.tezosagora.org/t/announcing-the-edo-release/2443">Edo</a> was activated on <a href="https://tzstats.com/election/25">13 February 2020</a> (<a href="https://tzstats.com/1343489">block height 1,343,489</a>; <a href="https://tzstats.com/cycle/328">cycle 328</a>; <a href="https://tezos.gitlab.io/protocols/008_edo.html">changelog</a>).</li>
</ul>
<p>We are pleased to announce of 2021 Quarter 2 that:</p>
<ul>
<li><a href="https://research-development.nomadic-labs.com/florence-our-next-protocol-upgrade-proposal.html">Florence</a> was activated on <a href="https://tzstats.com/election/27">11 May 2021</a> (<a href="https://tzstats.com/1466368">block height 1,466,368</a>; <a href="https://tzstats.com/cycle/357">cycle 357</a>; <a href="https://tezos.gitlab.io/protocols/009_florence.html">changelog</a>).</li>
<li><a href="https://forum.tezosagora.org/t/announcing-granada/3183">Granada</a> was proposed (<a href="https://tzstats.com/election/29">ongoing election</a>; <a href="https://tezos.gitlab.io/protocols/010_granada.html">changelog</a>) — and approved as this article went to press on July 20.</li>
</ul>
<h3 id="the-florence-upgrade"><a class="toclink" href="#the-florence-upgrade">The Florence upgrade</a></h3>
<p>You can find out more about the Florence upgrade here:</p>
<ul>
<li><a href="https://medium.com/tqtezos/florence-tezos-sixth-protocol-upgrade-goes-live-bringing-further-gas-optimizations-more-557836cc63ba">Florence, Tezos’ Sixth Protocol Upgrade Goes Live, Bringing Further Gas Optimizations <span class="amp">&</span> More</a> (on <span class="caps">TQ</span> Tezos).</li>
<li><a href="https://forum.tezosagora.org/t/florence-the-latest-tezos-upgrade-is-live/3113">Florence, the latest Tezos upgrade, is <span class="caps">LIVE</span></a></li>
<li><a href="https://tezos.gitlab.io/protocols/009_florence.html">Florence changelog</a> (on Tezos Developer Resources).</li>
</ul>
<p>Substantive Florence upgrades include:</p>
<ul>
<li><a href="https://research-development.nomadic-labs.com/sound-and-fast-gas-monitoring-with-saturation-arithmetic.html">Gas optimisations using saturation arithmetic</a> (so you get more smart contract execution for your gas).
Our benchmarks indicate a <a href="https://research-development.nomadic-labs.com/sound-and-fast-gas-monitoring-with-saturation-arithmetic.html#what-are-the-gains">tenfold speedup of gas computation, and a 35% speedup of the execution cycle of the smart contract Michelson interpreter in Florence overall</a>.</li>
<li>The maximum operation size is doubled, meaning that the maximum length of a smart contract is doubled. </li>
<li><a href="https://research-development.nomadic-labs.com/tezos-calling-convention-migrating-from-breadth-first-to-depth-first-order-bfs-to-dfs.html">Tezos calling convention has migrated from Breadth-First to Depth-First Order (<span class="caps">BFS</span> to <span class="caps">DFS</span>)</a>, making smart contract development more intuitive and less liable to bugs (see also a <a href="https://gitlab.com/tezos/tezos/-/merge_requests/2420">discussion of the code design</a>).</li>
<li>Florence drops the Test Chain feature of the Economic Protocol, because in practice it wasn’t needed.</li>
</ul>
<p>For more reading see:</p>
<ul>
<li><a href="https://tezos.gitlab.io/protocols/009_florence.html">the detailed Florence changelog</a>, and</li>
<li><a href="https://research-development.nomadic-labs.com/florence-our-next-protocol-upgrade-proposal.html">an accessible discussion</a>. </li>
</ul>
<h3 id="the-granada-upgrade-proposal"><a class="toclink" href="#the-granada-upgrade-proposal">The Granada upgrade proposal</a></h3>
<p>As per a <a href="https://research-development.nomadic-labs.com/announcing-granada.html">detailed blogpost on the Granada upgrade proposal</a>, Granada proposes the following changes:</p>
<ul>
<li>
<p>A <a href="https://research-development.nomadic-labs.com/faster-finality-with-emmy.html">switch from Emmy+ consensus to Emmy*</a>. Emmy* halves block time from 60 to 30 seconds (doubling transactions per second), increases the number of endorsement slots from 32 to 256 (increasing stability and participation), and provides a special fast consensus path for when the network is operating normally. All this makes Emmy* significantly faster than Emmy+. </p>
</li>
<li>
<p><a href="https://gitlab.com/tzip/tzip/-/blob/master/drafts/current/draft-liquidity_baking.md">Liquidity Baking</a> promotes low-slippage exchange between tez and other currencies using wrapped tokens.</p>
</li>
<li>
<p>The improvements to gas consumption continue. Empirically, we observed gas consumption decrease by a factor of three to six in the execution of already-deployed contracts.</p>
</li>
<li>
<p>Regrettably, Granada also contains a <a href="https://research-development.nomadic-labs.com/granada-comparisons-bug.html">non-critical bug</a>. We expect this to be corrected in the following “H”-named upgrade proposal. In the meantime, we will provide a linting tool to help developers detect the bug-affected pattern in any new smart-contract code. </p>
</li>
</ul>
<p>For further reading, see:</p>
<ul>
<li><a href="https://tezos.gitlab.io/protocols/010_granada.html">The Granada changelog</a>. </li>
<li><a href="https://research-development.nomadic-labs.com/announcing-granada.html">The Granada announcement</a>. </li>
</ul>
<h3 id="tenderbakenet"><a class="toclink" href="#tenderbakenet">Tenderbakenet</a></h3>
<p>We plan to <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">upgrade our consensus algorithm to Tenderbake</a> in the near future.
To this end, we are pleased to announce <strong>Tenderbakenet</strong>, an experimental Tezos blockchain based on Tenderbake.</p>
<ul>
<li>See the <a href="http://tenderbakenet.tzkt.io/">TzKT block explorer for Tenderbakenet</a> (many thanks to <a href="https://baking-bad.org/">Baking Bad</a>), and</li>
<li><a href="https://gitlab.com/nomadic-labs/tezos/-/blob/tenderbake-testnet/tenderbakenet.md">the instructions for joining Tenderbakenet</a>. </li>
</ul>
<p>Further information is in <a href="https://medium.com/tqtezos/rapid-innovation-tezos-tenderbake-testnet-spawned-9168da6599d9">an article on “Rapid Innovation: Tezos Tenderbake Testnet Spawned”</a>.</p>
<h2 id="testing"><a class="toclink" href="#testing">Testing</a></h2>
<p>We continue to improve our <a href="https://tezos.gitlab.io/developer/tezt.html">home-grown Tezt framework</a>.
One substantive new feature is the ability to <a href="https://gitlab.com/tezos/tezos/-/merge_requests/2934">run tests on remote hosts (see merge request)</a>. </p>
<p>How is this substantive?
Tezt was originally a tool for running <a href="https://research-development.nomadic-labs.com/measuring-test-coverage.html#test-coverage-for-the-michelson-interpreter">a test suite</a> of local unit and integration tests on your local machine.
This new functionality makes it convenient to run tests on remote hosts; Tezt-ing by remote control, so to speak.
This opens up possibilities to administer unit and integration Tezts <em>at scale</em> on clusters of remote nodes.</p>
<p>We’re also investing heavily in <a href="https://medium.com/criteo-engineering/introduction-to-property-based-testing-f5236229d237">property-based testing</a> (see also <a href="https://research-development.nomadic-labs.com/progress-report-on-the-verification-of-liquidity-baking-smart-contracts.html">our previous blog post</a>).
For example:</p>
<ul>
<li>
<p>We have completed a migration from crowbar property-based testing framework to the more mature <a href="https://c-cube.github.io/qcheck/">QCheck framework</a>.
In particular, our friends at Tweag have developed a new version <a href="https://github.com/c-cube/qcheck/pull/109">QCheck2</a>, which includes <strong>integrated shrinking</strong>, which means that <a href="https://web.archive.org/web/20210721090916/https://www.tweag.io/blog/2021-07-21-qcheck2-integrated-shrinking/">QCheck2 automatically searches for minimal <strong>counterexamples</strong></a>,<sup id="fnref:counter"><a class="footnote-ref" href="#fn:counter">4</a></sup> without requiring the programmer to write a shrinker by hand.<sup id="fnref:shrinking"><a class="footnote-ref" href="#fn:shrinking">5</a></sup></p>
</li>
<li>
<p>In March 2021 Valentin Chaboche started an internship on enabling large-scale light-weight specification and testing using QCheck.
His development consists of </p>
<ul>
<li>a <a href="https://github.com/ocaml-ppx/ppx_deriving">ppx to derive generators</a> and </li>
<li>a ppx to transform annotation-based function specifications to property-based tests.</li>
</ul>
<p><strong>ppx</strong> is an OCaml metaprogramming framework, and a <strong>generator</strong> is a program to generate test cases e.g. for QCheck or QCheck2; so that makes a <em>ppx to derive generators</em> an <em>“OCaml metaprogram for a programmer to programmatically generate generators to test code the programmer generated”</em>.
Similarly a <em>ppx to transform specifications to property-based tests</em> means more metaprogramming, for even more automatic test generation.
All in all, this means: lots and <em>lots</em> of new tests, many of which are generated by the computer itself with minimal or no further programmer input.</p>
</li>
</ul>
<p>To most effectively take advantage of these advances in tooling, we’re refactoring our code to expose and thus test interfaces in the codebase (more details in our <a href="https://gitlab.com/tezos/tezos/-/milestones/29">official “better testing through refactoring” milestone</a>).
We’re concentrating on hardening a set of critical code layers in the shell: the peer-to-peer layer, the distributed database, and the mempool. </p>
<p>This effort is complemented by the development of an <em>adversarial testing tool</em> by <a href="https://functori.com">Functori</a>.
The tool, currently in design, is intended to enable the specification and execution of network attack scenarios, thus enabling early detection of security issues in <a href="https://research-development.nomadic-labs.com/announcing-octez.html">Octez</a> (an implementation of the Tezos blockchain in OCaml).<sup id="fnref:pentest"><a class="footnote-ref" href="#fn:pentest">6</a></sup></p>
<p>Finally, the better to check the coverage of our test suite, we’re working on improving our usage of test coverage reports.
This happens mainly through the switch to a newer version of <a href="http://aantron.github.io/bisect_ppx/">bisect_ppx</a> and forthcoming usability and automation improvements in our <a href="https://docs.gitlab.com/ee/ci/">continuous integration environment</a>.</p>
<h2 id="interviews-and-papers"><a class="toclink" href="#interviews-and-papers">Interviews and Papers</a></h2>
<p>We are delighted to report that:</p>
<ul>
<li>On 9 April our head of adoption Hadrien Zerah was <a href="https://partenaires.lepoint.fr/lirresistible-ascension-des-cryptomonnaies/les-cryptomonnaies-nouvel-instrument-de-souverainete-monetaire">interviewed in Le Point magazine (in French)</a> about <em>“L’irrésistible Ascension des Cryptomonnaies”</em> (the irresistable rise of cryptocurrencies).</li>
<li>On 13 May, <a href="https://tezos.org.ua/en/blog/hadrien-zerah-spoke-about-mass-adoption-of-tezos">Tezos Ukraine interviewed Hadrien Zerah</a>.</li>
<li>You may be interested in this interview with <a href="https://medium.com/tqtezos/community-conversations-behind-the-scenes-with-vincent-botbol-research-and-development-architect-45614e2b0c34">Vincent Botbol, Research and Development Architect at Nomadic Labs</a>.</li>
</ul>
<p>We commissioned <a href="https://research-development.nomadic-labs.com/files/2021-01-report-private-amendment.pdf">a report by <span class="caps">INRIA</span> on “Possible evolutions of the voting system in Tezos”</a>, which was released on April 14. See also the <a href="https://forum.tezosagora.org/t/announcing-the-report-possible-evolutions-of-the-voting-system-in-tezos/3002">discussion on Tezos Agora</a>.</p>
<p>We are also delighted to announce that:</p>
<ul>
<li><a href="https://binsec.github.io/new/publication/1970/01/01/nutshell-ccs-21.html">Richard Bonichon <em>et al’s</em> paper on “Search-Based Local Black-Box Deobfuscation: Understand, Improve, and Mitigate”</a> (see also <a href="https://rbonichon.github.io/papers/xyntia-ccs21.pdf">author’s pdf</a>) was accepted to the <a href="https://www.sigsac.org/ccs/CCS2021/"><span class="caps">ACM</span> <span class="caps">CCS</span> 2021 conference (15-19 November 2021)</a>. Well done Richard!</li>
<li><a href="http://www.secrypt.org/Abstract.aspx?idEvent=obn3ztrWzrs=#Area0FullPapers">Richard Bonichon <em>et al’s</em> paper on “Program Protection Through Software-Based Hardware Abstraction”</a> (see also <a href="https://rbonichon.github.io/papers/sbha-secrypt21.pdf">author’s pdf</a>) was accepted and presented at <a href="http://www.wikicfp.com/cfp/servlet/event.showcfp?eventid=115969&copyownerid=45217">Secrypt 2021 (6-8 July 2021)</a>. Well done again Richard!</li>
<li><a href="https://arxiv.org/abs/2001.11965">Lăcrămioara Astefănoaei, Pierre Chambart, Eugen Zălinescu, <em>et al’s</em> paper on “Tenderbake - A Solution to Dynamic Repeated Consensus for Blockchains”</a> was <a href="https://youtu.be/aoT39V-ySTc">presented</a> at the <a href="https://scfab.github.io/2021/schedule.html">Fourth International Symoposium on Foundations and Applications of Blockchain (<span class="caps">FAB</span>‘21)</a>. Well done Lacra, Pierre, and Eugen!</li>
</ul>
<h2 id="nl-research-seminars"><a class="toclink" href="#nl-research-seminars"><span class="caps">NL</span> research seminars</a></h2>
<p>Our series of <a href="https://www.youtube.com/watch?v=2jd-1VxzmQ0&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7">Nomadic Labs research seminars</a> continues apace.
In Q2 we saw:</p>
<ol>
<li><a href="https://www.youtube.com/watch?v=BGKaq9I9KKw&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=11">Helmholtz - A Verifier for Tezos Smart Contracts Based on Refinement Types</a> (20 April 2021)</li>
<li><a href="https://www.youtube.com/watch?v=KXiyXApMyxM&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=12">Information Extraction from Graphs and the TezQuery Tool</a> (13 April 2021)</li>
<li><a href="https://www.youtube.com/watch?v=KXiyXApMyxM&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=13">Verifiable Delay Functions and Groups of Unknown Order</a> (27 April 2021)</li>
<li><a href="https://www.youtube.com/watch?v=KXiyXApMyxM&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=14">On Oracles and Contract Modules</a> (11 May 2021)</li>
<li><a href="https://www.youtube.com/watch?v=KXiyXApMyxM&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=15"><span class="caps">TLA</span>+ Formal Specification of Bootstrapping</a> (25 May 2021)</li>
<li><a href="https://www.youtube.com/watch?v=KXiyXApMyxM&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=16">Package Tezos as a MirageOS Unikernel</a> (08 June 2021)</li>
<li><a href="https://www.youtube.com/watch?v=KXiyXApMyxM&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=17">Prototype of a Typical Smart Contract Agency</a> (22 June 2021)</li>
</ol>
<p><a href="https://www.youtube.com/playlist?list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7">Click here for full list of talks</a>.</p>
<h2 id="tezos-is-3"><a class="toclink" href="#tezos-is-3">Tezos is 3</a></h2>
<p>Tezos is three years old!
Specifically, June 30 2021 — just at the end of Q2 2021 — was the three-year anniversary of the Tezos genesis block, which was baked on <a href="https://tzstats.com/0">June 30 2018</a>.
See the <a href="https://twitter.com/i/status/1410268491048001538">video on Twitter</a>.</p>
<p><a name="prochaine"></a></p>
<h2 id="a-la-prochaine"><a class="toclink" href="#a-la-prochaine">À la prochaine</a></h2>
<p>And that’s what we’ve been up to in Quarter 2 of 2021: three months of Nomadic Labs building and testing software and extending public understanding and adoption of blockchain technology.
Thanks for reading, and do check in again for the next Meanwhile for Quarter 3 of 2021.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:two">
<p>We have two teams: <em>adoption</em> cultivates relationships with new users; <em>support</em> helps existing users to implement solutions. A third team <a href="#train">focusses on training</a>. <a class="footnote-backref" href="#fnref:two" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:baker">
<p>A <strong>baker</strong> is a block validator on the Tezos blockchain. An <strong>institutional baker</strong> is a corporate institution that sets up one or more validating nodes on the Tezos blockchain. <a href="https://tzstats.com/docs/guide/page/bakers/">More information on bakers here</a>. <a class="footnote-backref" href="#fnref:baker" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:library">
<p>Technically it’s a library of cryptographic primitives and other functions, packaged as <a href="https://tezos.gitlab.io/api/odoc/_html/tezos-protocol-environment/Tezos_protocol_environment/index.html">an OCaml module</a>. <a class="footnote-backref" href="#fnref:library" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:counter">
<p>Inputs that break correctness assertions, i.e. inputs that suggest errors either in the program, or in the programmer’s understanding of their program’s correctness. Also see next footnote. <a class="footnote-backref" href="#fnref:counter" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:shrinking">
<p>Some context: in property-based testing, the programmer states desired correctness properties and then leverages the computer itself to generate (thousands, tens of thousands, or even millions of) test cases for their code with respect to their desired correctness properties.</p>
<p>In practice this is very effective — <em>but</em> it is important for subsequent debugging that counterexamples be small and so easy to understand;
for a given bug, a small and precise counterexample to trigger it is far easier to trace than a large and redundant one.
Thus, while a property-based testing tool may find a counterexample, it is the job of a <strong>shrinker</strong> to input a possibly large counterexample and find a small one that a programmer can quickly dispatch.</p>
<p>It’s suboptimal if a programmer has to write shrinkers by hand, since then we may just be replacing the problem of writing bug-free code with the problem of writing bug-free <em>shrinkers</em> for code!
This might still be a worthwhile tradeoff, but it would be better and also safer if it could happen automagically.
This is the burden of which QCheck2 relieves the programmer, relative to QCheck, and doing this is possible because shrinking tends to be a heuristic process: an “integrated shrinker” is a bundle of heuristics which, in practice, tends to automatically generate effective shrinkers that find small counterexamples from large ones. <a class="footnote-backref" href="#fnref:shrinking" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:pentest">
<p>Think: a <a href="https://en.wikipedia.org/wiki/Penetration_test">penetration testing tool</a>, applied to give us efficient white-hat hacker testing of our releases. <a class="footnote-backref" href="#fnref:pentest" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
</ol>
</div>Three questions to Nomadic Labs interns — Antonio Locascio2021-07-21T11:00:00+02:002021-07-21T11:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-07-21:/three-questions-to-nomadic-labs-interns-antonio-locascio.html<p>Short interview with Antonio Locascio, our new intern</p><p>Each year we host around half-a-dozen interns (<em>stagiaires</em>), for periods of two to six months.
These young men and women are usually studying or have just finished studying for undergraduate or Master’s degrees, and are eager to gain experience in the blockchain industry.
Each has a unique story to tell about their individual interest in this technology.</p>
<p>In this blogpost, we will ask three questions of one of our current interns: <strong>Antonio Locascio</strong> (and a couple of questions of his mentors at Nomadic Labs).</p>
<p>Antonio — so happy to have you with us!
We hope you’re having a wonderful and educational period with Nomadic Labs.
We’d love to hear a bit about you and your activities <em>chez nous</em> …</p>
<h1 id="questions-for-antonio"><a class="toclink" href="#questions-for-antonio">Questions for Antonio</a></h1>
<h2 id="1-please-present-yourself-and-your-academic-background"><a class="toclink" href="#1-please-present-yourself-and-your-academic-background">1. Please present yourself and your academic background</a></h2>
<p>I graduated last year as a <a href='https://en.wikipedia.org/wiki/Licentiate_(degree)#Argentina'>Licentiate</a> (a five-year undergraduate degree) in Computer Science from the <a href='https://unr.edu.ar/'>Universidad Nacional de Rosario</a>, in Argentina (where I’m from).
My main academic interests since my time at university are in the fields of functional programming, programming languages and type systems.
For my Licentiate thesis, I worked on a bidirectional type-and-effect system for a core language with algebraic effects, under the supervision of Mauro Jaskelioff and Exequiel Rivas.</p>
<p>Towards the end of my studies, I became quite interested in the field of formal verification, although I didn’t have much first hand experience in it. This was one of the reasons I was so keen on doing this internship, as it was a great opportunity to learn more about this topic while working on real-world problems.</p>
<p>After graduating, I was still undecided on which career path to take regarding whether to continue in academia or go straight to industry. I figured that doing an internship could help me with this decision, and there was hardly a better place to do so than in Nomadic Labs. Here, I would get a taste of what working in an industrial setting is like while still being exposed to, and taking part in, the most recent developments in the academic fields.</p>
<h2 id="2-tell-us-more-about-your-internship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months"><a class="toclink" href="#2-tell-us-more-about-your-internship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months">2. Tell us more about your internship: main subject, who is your mentor, what you have learned and especially, what surprised you the most within these months?</a></h2>
<p>My mentors for my internship here at Nomadic Labs are Marco Stronati, Germán Delbianco and Victor Dumitrescu.
I’ve been working on verifying OCaml code using the <a href='https://fstar-lang.org/'>F* language</a>. F* is a <em>hybrid</em> verification-oriented programming language.
This means that it not only has the expressivity of interactive theorem provers based on dependent types, but thanks to its <span class="caps">SMT</span> support it can also automatically prove many properties.</p>
<p>The goal of my internship is to evaluate a verification workflow consisting of three steps.
The first one is to model a piece of OCaml code in F*, specifying and proving interesting properties that it should satisfy.
When this effort is completed, the next step is to extract a certified OCaml implementation from the model, with which to replace the original implementation.
The third step, perhaps the most novel, is to extract the specification itself from the model, as Property Based Tests for the OCaml code.
We consider this last step very important, as it helps to close the <em>verification gap</em>, i.e. the semantic difference between the verified model and the actual OCaml code.</p>
<p>Concretely, I’ve first applied this workflow to the implementation of <a href='https://tezos.gitlab.io/alpha/sapling.html'>Sapling</a>, a protocol used in Tezos for enabling privacy-preserving transactions.
The main verification effort for this initial case study was focused on its storage, which consists of a special type of Merkle tree.
After finishing with it, I’ve moved on to the <a href='https://gitlab.com/nomadic-labs/privacy-team/-/tree/master/rollup'><span class="caps">ZK</span>-Rollup project</a>, which proposes a Layer 2 scalability solution with minimal impact on the main chain.
Before starting its verification, I’ve been working on a prototype OCaml implementation that is helpful for tinkering with its design.
The Rollup’s storage is also a Merkle tree, so the experience from the Sapling model will definitely come in handy for this new verification effort.</p>
<p>I’ve learned a lot during these months: from verification and testing techniques, to code reviewing and good programming practices.
All of those things point to what probably surprised me the most for now, which is the importance given to program correctness here at Nomadic Labs.
Although it should be expected for a company working on critical code, it was great to see that correctness was emphasized at every step of the pipeline, which is sadly not the case very often.</p>
<h2 id="3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-internship"><a class="toclink" href="#3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-internship">3. Why did you choose to become a part of the blockchain ecosystem and Nomadic labs? What are your plans after completing this internship?</a></h2>
<p>Although I didn’t really know much about blockchain, the nature of it being an emerging technology with many interesting problems still to be solved compelled me.
This field is perfect for verification, as there are plenty of important properties about their implementation that are worth proving.
As I explained above, the appeal of Nomadic Labs is its rigorous scientific background, which guarantees an opportunity to learn about interesting new developments and ideas.
After my internship, I plan to continue working as a developer on cutting edge projects such as this one.</p>
<h1 id="questions-for-antonios-nomadic-labs-mentors-marco-stronati-and-german-delbianco"><a class="toclink" href="#questions-for-antonios-nomadic-labs-mentors-marco-stronati-and-german-delbianco">Questions for Antonio’s Nomadic Labs mentors Marco Stronati and Germán Delbianco:</a></h1>
<h2 id="what-is-your-input-to-the-work-of-antonio"><a class="toclink" href="#what-is-your-input-to-the-work-of-antonio">What is your input to the work of Antonio?</a></h2>
<p>As Antonio’s mentors, our role is to provide guidance based on our combined experience working on the development of the Tezos Economic Protocol, on mechanized software development and verification, and in the toolchain itself.
We plant seeds, in the form of suggesting relevant academic literature or related projects; we brainstorm ideas together; we provide feedback on his contributions; and answer his questions. Occasionally, we also warn him of known pitfalls, and of <a href='https://en.wikipedia.org/wiki/Here_be_dragons'><em>Here be dragons</em></a>.</p>
<p>That said, we are pleased by the fact that Antonio has hit the ground running on this project, and by the independence he has shown.
In order to implement the objectives we had devised together, he not only succeeded in learning how to verify programs in F*, but also he quickly realized how to improve the toolchain so that we could take the most out of this experiment.
This makes this work relevant not just to us at Nomadic Labs, or to the Tezos ecosystem at large, but eventually to the F* community as well. </p>
<h2 id="why-is-the-topic-of-this-internship-important"><a class="toclink" href="#why-is-the-topic-of-this-internship-important">Why is the topic of this internship important?</a></h2>
<p>Making verification techniques scale up to a large industrial, cutting edge, project is a Herculean task.
In our case, one of the great hurdles is tightening the verification gap between a mechanization in F*, and the <em>real-world</em> Tezos implementation in OCaml.
In a large and complex codebase like ours, there is inevitably a semantic distance between the abstract models we build for the verification of critical components of the Tezos Economic Protocol, such as Sapling or <span class="caps">ZK</span>-rollups, and the nitty-gritty of their implementation.</p>
<p>In this project we are indeed interested in reducing this semantic gap, by recovering specifications from the F* mechanization, to validate both the extracted OCaml code and the (different) implementations in the codebase. </p>
<p>Extracting the specification as property-based tests automatically from a mechanized implementation (technically, from dependently-typed signatures) is a (to the best of our knowledge) novel contribution of this internship.
Usually, tools like F* (or Coq, or Agda) are designed so that specs and proofs are erased from type signatures, and thus they do not feature in the extracted code — they are lost!
Antonio’s solution to recover them is quite elegant, as it relies on two existing F* features — meta-programming and tactics — to extract the definitions and the specs as QCheck tests. Hence, this approach doesn’t require modifying how the extraction mechanism in F* works, but only some tweaking of the <a href='https://fstar-lang.org/papers/metafstar'>Meta-F* framework</a>.</p>
<p>In an ideal world, we would like to have a development cycle which efficiently integrates and interweaves concurrent verification and development efforts.
This requires developing common specification styles and languages that can be shared by developers and <strong>verifiers</strong><sup id="fnref:verifiers"><a class="footnote-ref" href="#fn:verifiers">1</a></sup>.
This internship provides a plausible approach: by extracting specs from F* and translating them directly to QCheck tests, we can ensure that the specs and definitions in both F* and OCaml — that is both in Proofs and Tests — are two correct reifications of the same concepts, increasing our confidence in the verification process.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:verifiers">
<p>Formal methods/verification experts, or anyone who gets a kick out of writing mechanized proofs. <a class="footnote-backref" href="#fnref:verifiers" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Three questions to Nomadic Labs interns — Tianchi Yu2021-07-19T14:00:00+02:002021-07-19T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-07-19:/three-questions-to-nomadic-labs-interns-tianchi-yu.html<p>Short interview with Tianchi Yu, our new intern</p><p>Each year we host around half-a-dozen interns (<em>stagiaires</em>), for periods of two to six months.
These young men and women are usually studying or have just finished studying for undergraduate or Master’s degrees, and are eager to gain experience in the blockchain industry.
Each has a unique story to tell about their individual interest in this technology.</p>
<p>In this blogpost, we will ask three questions of one of our current interns: <strong>Tianchi Yu</strong> (and a couple of questions of his mentors at Nomadic Labs).</p>
<p>Tianchi — so happy to have you with us!
We hope you’re having a wonderful and educational period with Nomadic Labs.
We’d love to hear a bit about you and your activities <em>chez nous</em> …</p>
<h1 id="questions-for-tianchi"><a class="toclink" href="#questions-for-tianchi">Questions for Tianchi</a></h1>
<h2 id="1-please-present-yourself-and-your-academic-background"><a class="toclink" href="#1-please-present-yourself-and-your-academic-background">1. Please present yourself and your academic background</a></h2>
<p>My name is Tianchi <span class="caps">YU</span>. I have been studying in the Master Program on <a href='https://www.ip-paris.fr/en/education/masters/computer-science-program/master-year-2-cyber-physical-systems'>Cyber-Physical Systems</a> at the <a href='https://www.polytechnique.edu/'>École Polytechnique</a> since September 2020.
From March 2020 to August 2020 I studied in the Computer Science Department at <a href='https://www.epfl.ch/en/'><span class="caps">EPFL</span></a>, and from September 2018 to February 2020 I studied in the Ingénieur Program at <a href='https://www.isae-supaero.fr/'><span class="caps">ISAE</span>-<span class="caps">SUPAERO</span></a> in Toulouse.
Before that, I did the Engineering Bachelor Program at <a href='https://www.seu.edu.cn/english/'>Southeast University</a> in China.</p>
<h2 id="2-tell-us-more-about-your-internship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months"><a class="toclink" href="#2-tell-us-more-about-your-internship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months">2. Tell us more about your internship: main subject, who is your mentor, what you have learned and especially, what surprised you the most within these months?</a></h2>
<p>The Tezos blockchain has a high-level bytecode smart contract language called <a href='https://tezos.gitlab.io/active/michelson.html'>Michelson</a>.
Michelson programs consume gas, which is an abstract resource designed to bound smart contract computation and thus (amongst other things) incentivize efficient use of on-chain computation.
My internship at Nomadic labs is to optimize Michelson programs to consume the least gas possible, i.e. to do their jobs while consuming minimal on-chain resources. </p>
<p>Specifically we are studying <a href='https://en.wikipedia.org/wiki/Superoptimization'>superoptimization</a> (finding global program optimizations which might be missed by a smaller and simpler search for local optimizations).
We’re aiming for an <span class="caps">AI</span>-based method to find the optimal bytecode in a fully blackbox way (i.e. without user intervention). </p>
<p>My mentors are Richard Bonichon and Yann Régis-Gianas.
They have been very dedicated and supportive.</p>
<p>Over the past few months I’ve gained a deeper understanding of Tezos in general, the Michelson language in particular, and I’ve learned about OCaml and advanced skills in research and development as practiced in a real day-to-day industrial setting.
I am impressed by the scalable and robust structure of the Tezos ecosystem; by the efficient, continuous contributions of programmers at Nomadic Labs; and by the exceptional competence and dedication of the teams that work behind the scenes to develop Tezos.
In short, for me this internship is a holistic experience: I’ve learned about programming, how to be a programmer, how to be a teacher, and how a top-quality open-source community-driven industrial project can (and should!) be run.
It’s really been an inspiration for me. </p>
<h2 id="3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-internship"><a class="toclink" href="#3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-internship">3. Why did you choose to become a part of the blockchain ecosystem and Nomadic labs? What are your plans after completing this internship?</a></h2>
<p>I have always been curious about how blockchains work and I am eager to be one part of the contributions to it. There are many interesting topics to explore.
For example: optimization, consensus, verification, and also security.
Functional programming is a very skillful and interesting approach, and this led me to Tezos and Nomadic Labs. </p>
<p>After this internship, I plan to complete my last year of the Ingénieur Program.
After that I plan to focus on Computer Science and Optimization, hopefully as a programmer or researcher.
I hope that I will have the chance to get a PhD or do more industrial research. </p>
<h1 id="questions-for-tianchis-nomadic-labs-mentors-richard-bonichon-and-yann-regis-gianas"><a class="toclink" href="#questions-for-tianchis-nomadic-labs-mentors-richard-bonichon-and-yann-regis-gianas">Questions for Tianchi’s Nomadic Labs mentors Richard Bonichon and Yann Régis-Gianas:</a></h1>
<h2 id="what-is-your-input-to-the-work-of-tianchi"><a class="toclink" href="#what-is-your-input-to-the-work-of-tianchi">What is your input to the work of Tianchi?</a></h2>
<p>This internship topic is quite involved and can have many ramifications, so our main goal is to guide Tianchi to get a minimal viable prototype (<span class="caps">MVP</span>) for an <span class="caps">AI</span>-driven superoptimizer.
This includes weekly debrief meetings, answering his questions, and in general steering him towards programming a first prototype while listening to his inputs.</p>
<h2 id="why-is-the-topic-of-this-internship-important"><a class="toclink" href="#why-is-the-topic-of-this-internship-important">Why is the topic of this internship important?</a></h2>
<p>Code optimization is a critical component of practical compilers.
For smart contracts and Michelson bytecode, optimization saves not only time but also crypto-fees through lower gas consumption.</p>
<p>Normal compilers balance speed of code generation against speed of the code generated.
But for smart contracts, on-chain computation is so much more valuable than off-chain computation, that almost absolute priority can be given to optimizing bytecode.
This seems an excellent application for superoptimization, since (to simplify only slightly) its job is to exhaustively search for the absolute most efficient code possible.</p>
<p>Tianchi’s work is an initial and interesting step towards achieving this in the Tezos ecosystem, with a way to optimize smart contracts <em>a posteriori</em> in a compiler-agnostic manner.</p>Three questions to Nomadic Labs interns — Corentin Calmels2021-07-15T10:00:00+02:002021-07-15T10:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-07-15:/three-questions-to-nomadic-labs-interns-corentin-calmels.html<p>Short interview with Corentin Calmels, our new intern</p><p>Each year we host around half-a-dozen interns (<em>stagiaires</em>), for periods of two to six months.
These young men and women are usually studying or have just finished studying for undergraduate or Master’s degrees, and are eager to gain experience in the blockchain industry.
Each has a unique story to tell about their individual interest in this technology.</p>
<p>In this blogpost, we will ask three questions of one of our current interns: <strong>Corentin Calmels</strong> (and a couple of questions of his mentor at Nomadic Labs).</p>
<p>Corentin; so happy to have you with us!
We hope you’re having a wonderful and educational period with Nomadic Labs.
Please, tell us a bit about yourself and your activities <em>chez nous</em> …</p>
<h1 id="questions-for-corentin"><a class="toclink" href="#questions-for-corentin">Questions for Corentin</a></h1>
<h2 id="1-please-present-yourself-and-your-academic-background"><a class="toclink" href="#1-please-present-yourself-and-your-academic-background">1. Please present yourself and your academic background</a></h2>
<p>My name is Corentin Calmels.
I’m 20 years old and from Paris.
I speak fluent French and English, and am studying German and Japanese.</p>
<p>I am a student in <a href='https://www.hec.edu/en'><span class="caps">HEC</span> Paris business school</a> <a href="https://www.hec.edu/en"><span class="caps">HEC</span> Paris business school</a>.
I completed the <a href='https://www.bloomberg.com/professional/product/bloomberg-market-concepts/'> Bloomberg Certificate of Finance</a> and I would like to specialize in finance in my future career.
I’m doing a summer internship at Nomadic Labs, from June to early August 2021. </p>
<p>I discovered blockchain through newspaper headlines, but I wanted to go deeper and link my studies with blockchain, so I got a job in May with <a href='https://www.linkedin.com/company/association-hec-bourse/mycompany/'><span class="caps">HEC</span> Bourse</a>(the <span class="caps">HEC</span> finance association) as a cryptocurrency columnist, and applied for a summer internship at Nomadic Labs.
In so doing, I discovered an incredible technology.</p>
<h2 id="2-tell-us-more-about-your-internship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months"><a class="toclink" href="#2-tell-us-more-about-your-internship-main-subject-who-is-your-mentor-what-you-have-learned-and-especially-what-surprised-you-the-most-within-these-months">2. Tell us more about your internship: main subject, who is your mentor, what you have learned and especially, what surprised you the most within these months?</a></h2>
<p>I’m working at Nomadic Labs with the adoption team, under the supervision of Thibaut Chessé.
I’m learning about blockchain and adoption techniques, including about how to strengthen relationships with potential corporate adopters, and about analysis and presentation skills to help identify and solve potential adopters’ key needs.</p>
<p>I also study DeFi (Decentralized Finance), which I believe will be the core of tomorrow’s finance.
Blockchain itself is a key DeFi technology, and although DeFi is quite recent, I have big expectations for the field and it’s exciting to see it grow day by day. </p>
<p>My role is to support the adoption team.
I am in daily contact with companies and public clients to discuss and develop potential applications of the Tezos blockchain.
For instance, one of our recent adopters is McLaren Racing, which <a href='https://www.mclaren.com/racing/partners/tezos/'>is developing a Formula 1 <span class="caps">NFT</span> (non-fungible token) platform on Tezos</a>.</p>
<p>Since I began the internship on 1 June 2021, my understanding of blockchain and crypto has greatly increased.
I’ve been exposed to a diversity of fields in blockchain: <a href='https://web.archive.org/web/20210324105601/https://www.ledgerinsights.com/french-cbdc-digital-currency-socgen-societe-generale-tezos-public-blockchain/'>central bank digital currencies</a>; euro stablecoins such as <a href='https://www.lugh.io'>Lugh</a>; tokenized and decentralized art marketplaces such as <a href='https://www.hicetnunc.xyz'>HicEtNunc</a>; and even sports collectibles implemented as NFTs (blockchain-based collectible fan cards!) for example <a href='ttps://web.archive.org/web/20210531125835/https://www.ledgerinsights.com/red-bull-racing-f1-team-signs-nft-deal-with-tezos/'>in car racing in cooperation with Redbull</a>.</p>
<p>It is astonishing how much blockchain has already impacted on our lives.
Working here has opened my eyes to the industry’s potential.</p>
<h2 id="3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-internship"><a class="toclink" href="#3-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs-what-are-your-plans-after-completing-this-internship">3. Why did you choose to become a part of the blockchain ecosystem and Nomadic labs? What are your plans after completing this internship?</a></h2>
<p>I’m a Blockchain enthusiast and I want to learn more about this state-of-the-art technology, which will soon be ubiquitous.
Who better to work with than the pioneers in the field? </p>
<p>Nomadic Labs is one of the most important companies in the Tezos ecosystem and is a key contributor to its development.
Like me, Nomadic Labs is based in Paris, and it’s wonderful to have a great blockchain company on my doorstep — all the more so because Nomadic Labs also has a richness of nationalities and cultural diversity. </p>
<p>Tezos itself has some unique features that I think will make it the blockchain of tomorrow — such as its unique self-amendment facility, and the on-chain governance — and this is even more reason to be involved in this project.
After my internship, I plan to complete my studies, and then get professionally involved in digitalization of finance through blockchain.</p>
<h1 id="questions-for-corentins-nomadic-labs-mentor-thibaut-chesse"><a class="toclink" href="#questions-for-corentins-nomadic-labs-mentor-thibaut-chesse">Questions for Corentin’s Nomadic Labs mentor Thibaut Chessé:</a></h1>
<h2 id="what-is-your-input-to-the-work-of-corentin"><a class="toclink" href="#what-is-your-input-to-the-work-of-corentin">What is your input to the work of Corentin?</a></h2>
<p>As Corentin’s internship supervisor, I help him to understand blockchain-related business issues, and provide him with analytical methods and knowledge about blockchain and its ecosystem.</p>
<h2 id="why-is-the-topic-of-this-internship-important"><a class="toclink" href="#why-is-the-topic-of-this-internship-important">Why is the topic of this internship important?</a></h2>
<p>It’s fair to say that industry accepts that blockchain is or could be a transformative technology.
But, a big part of the challenge to actually applying blockchain to capture value is understanding how to support adoption, communication, marketing, and business processes.</p>
<p>Think of the sailing ship: the business case for transporting cargos across oceans is clear, but you can still invent insurance, limited companies, shareholders, new kinds of advertising, and many other business innovations to get the most out of the technology. </p>
<p>Thus Corentin is studying business and marketing strategies for promoting the emerging market of blockchain-based business solutions, and thinking about how to systematically analyze the complexities and the opportunities that players from different industries face when developing blockchain innovations.
It’s a nascent field and the scope for business innovation is effectively unlimited. </p>Five questions to Nomadic Labs PhDs — Paul Laforgue2021-07-09T10:00:00+02:002021-07-09T10:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-07-09:/five-questions-to-nomadic-labs-phds-paul-laforgue.html<p>Short interview with Paul Laforgue, our new PhD student</p><p>At Nomadic Labs we are proud to create next-gen software … but we are even prouder to help create the next generation of software <em>scientists</em>!</p>
<p>So it’s our great pleasure to host and work with several PhD students, who bring unique perspectives on the technology which they help us to develop, and whose interest in blockchain we are happy to nurture and inform. </p>
<p>In this blogpost, we will ask five questions of one of our students, <strong>Paul Laforgue</strong> (and a couple of questions of his supervisors at Nomadic Labs).</p>
<p>Over to you Paul … no the mic is over here … that’s right, you’re on air!</p>
<h1 id="questions-for-paul"><a class="toclink" href="#questions-for-paul">Questions for Paul</a></h1>
<h2 id="1-please-present-yourself-and-your-academic-background"><a class="toclink" href="#1-please-present-yourself-and-your-academic-background">1. Please present yourself and your academic background</a></h2>
<p>I’m a PhD student working with the <a href="https://www.nomadic-labs.com/">Nomadic Labs <span class="caps">VERIF</span> team</a> and at the <a href="https://www.irif.fr/en/index"><span class="caps">IRIF</span> research laboratory</a> of the <a href="https://u-paris.fr/en/">Université de Paris</a>.</p>
<p>I’ve always been interested in programming language design and the use of proof-assistants to check correctness.
In 2015 I did my first research internship at <a href="https://www.chalmers.se/en/Pages/default.aspx">Chalmers University</a>, in which I was introduced to the <a href="https://en.wikipedia.org/wiki/Agda_(programming_language)">Agda proof assistant</a>.
In 2016, I started a Master’s degree at the Université de Paris under the supervision of Yann Régis-Gianas, who was at the Université at the time (and is now a programmer at Nomadic Labs).
<a href="https://hal.archives-ouvertes.fr/hal-01653261">Yann and I showed</a> how a programming language like OCaml can be extended with <a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.295.8056"><em>codatatypes</em> and <em>copatterns</em></a>; modern programming language constructs to define and handle infinite data structures.
Then in 2017 I did an internship in which I designed a type system for a subset of the R programming language at Northeastern University.</p>
<p>In October 2018 I started my job at Nomadic Labs in what is now called the <em>Shell team</em>.<sup id="fnref:shell"><a class="footnote-ref" href="#fn:shell">1</a></sup>
I mainly worked on the introduction of history modes, and metrics for the node.
I started my PhD thesis at the Université de Paris in late 2020.</p>
<p>During my time as a Nomadic programmer, I became acquainted with the Tezos OCaml implementation <a href="https://gitlab.com/tezos/tezos/">Octez</a> and with the challenges it has to face.
This background knowledge helped us define the goals we want this PhD thesis to tackle.</p>
<h2 id="2-tell-us-more-about-the-topic-of-your-phd-who-is-your-mentor-and-what-are-your-research-objectives"><a class="toclink" href="#2-tell-us-more-about-the-topic-of-your-phd-who-is-your-mentor-and-what-are-your-research-objectives">2. Tell us more about the topic of your PhD: who is your mentor, and what are your research objectives?</a></h2>
<p>Distributed, networked systems are now ubiquitous.
By <em>distributed system</em> I mean one consisting of multiple participants — usually in different locations — who coordinate by exchanging messages to achieve some goal. </p>
<p>Blockchains are distributed of course, but so are telecommunication networks, peer-to-peer systems, and distributed databases (e.g. in medicine and transportation).
Notice how all of these examples are also safety-critical!</p>
<p>Distributed systems tend to appear in applications that require the following features: </p>
<ul>
<li><strong>Reliability:</strong> a few faulty nodes won’t corrupt the system’s behaviour,</li>
<li><strong>Scalability:</strong> computing resources and scheduling methods can be dynamically adjusted, and </li>
<li><strong>Performance:</strong> incoming tasks can be distributed optimally amongst participants.</li>
</ul>
<p>But there’s a catch: programming concurrent and distributed systems is notoriously hard, and in particular it is very hard to predict all possible interactions between their components.</p>
<p><strong>Message-passing</strong> is a fundamental coordination mechanism used by participants in a distributed system.
We are developing a framework to specify and verify message-passing protocol specifications, and to validate an implementation against a given protocol.</p>
<p>In particular, we are developing a framework to specify and verify multiparty message-passing protocol specifications with the aim of making them executable and usable for conformance testing.
In this framework, written in Coq, protocols are defined as choreographies. </p>
<p><strong>Choreographies</strong> are global descriptions of distributed systems, expressing the composition of the expected interactions between their participants.
We want to specify and verify the desired properties<sup id="fnref:desired"><a class="footnote-ref" href="#fn:desired">2</a></sup> at the level of choreographies, and then produce certified executables by synthesising local behaviours of each participant that faithfully realise the communications given in the choreography, a mechanism known as <strong>endpoint projection</strong>.<sup id="fnref:chor"><a class="footnote-ref" href="#fn:chor">3</a></sup></p>
<p>My mentors at Nomadic Labs are Zaynah Dargaye, Thomas Letan and Yann Régis-Gianas.
My research directors from the <span class="caps">IRIF</span> research laboratory at Paris University are Giovanni Bernardi and Giuseppe Castagna.
All have theoretical and practical backgrounds on communication-based programming, which is essential for this project.
I’m lucky to work with such talented and expert people.</p>
<h2 id="3-what-is-the-value-added-and-innovative-side-of-your-phd-thesis-if-applicable-in-the-tezos-blockchain"><a class="toclink" href="#3-what-is-the-value-added-and-innovative-side-of-your-phd-thesis-if-applicable-in-the-tezos-blockchain">3. What is the value added and innovative side of your PhD thesis (if applicable, in the Tezos blockchain)?</a></h2>
<p>Tezos is open-source and relies on an open network, so anybody can build or profile its nodes and enter the network.
This is an important feature when we care for decentralization.
Today, two projects implement Tezos nodes: the OCaml implementation <a href="https://gitlab.com/tezos/tezos/">Octez</a> and the Rust implementation <a href="https://github.com/tezedge/tezedge">Tezedge</a>.
Also, numerous releases of the node are currently running on mainnet.
Miscommunication between the nodes may lead to critical issues,<sup id="fnref:network-split"><a class="footnote-ref" href="#fn:network-split">4</a></sup> thus to guarantee system health we must ensure that the nodes communicate well.</p>
<p>Also, it is important to build rigourous specifications, to avoid Tezos later encountering issues related with legacy code — as e.g. <a href="https://internationalfinance.com/legacy-systems-in-banks-explained/">core banking platforms are finding themselves struggling with now</a> (<a href="https://web.archive.org/web/20210115142535/https://internationalfinance.com/legacy-systems-in-banks-explained/">permalink</a>).<sup id="fnref:cobol"><a class="footnote-ref" href="#fn:cobol">5</a></sup>
In our case, the fact that the specification is executable offers benefits: it stays up-to-date with the implementations and can assist programmers when modifying the node.</p>
<h2 id="4-what-is-the-benefit-of-preparing-a-phd-and-spending-most-of-your-time-in-a-private-company-vs-a-university-lab"><a class="toclink" href="#4-what-is-the-benefit-of-preparing-a-phd-and-spending-most-of-your-time-in-a-private-company-vs-a-university-lab">4. What is the benefit of preparing a PhD and spending most of your time in a private company vs. a University lab?</a></h2>
<p>Both at Nomadic Labs and at the University, I am surrounded by people passionate about their fields, coming from different backgrounds and driven by curiosity.
However, there is a constructive difference in emphasis:</p>
<p>At the University people target the publication of academic papers, whilst at Nomadic Labs the target is the improvement of the Tezos implementation.
Nomadic programmers use their academic background to transform theoretical results from university research, into practical solutions.
That’s two distinct yet interdependent approaches to computer science research.
As a PhD student working at Nomadic Labs I get to study, participate in, and contribute to both approaches.</p>
<h2 id="5-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs"><a class="toclink" href="#5-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs">5. Why did you choose to become a part of the blockchain ecosystem and Nomadic labs?</a></h2>
<p>I believe that blockchain technology is promising — for decentralised finance of course, but also in other ways which I hope may be directly socially constructive as well: e.g. <a href="https://www.electis.app/static/landing/index.html">e-voting, such as electis</a>, <a href="https://sprucesystems.medium.com/announcing-tezos-profiles-own-and-control-your-creator-identity-ec0eddf9706d">decentralised identities</a>, and <a href="https://medium.com/tqtezos/tickets-on-tezos-part-1-a7cad8cc71cd">tickets</a>.</p>
<p>Tezos’ self-amendment mechanism makes it interesting, and it is also a fascinating project because it involves such a breadth of areas: whether you are interested in the peer-to-peer layer, in cryptography, in verification or consensus algorithms you will find something to dig in to and to improve the software.
On a more personal level I really appreciate all the energy coming from the community’s feedback.</p>
<p>I want to enhance my skills in the formal verification of distributed systems using proof assistants, so that by the end of the thesis can tackle industry challenges with the Verif team at Nomadic. </p>
<p>I would like to enrich my perspectives when coping with research problems, by learning from my mentors and studying their approaches to problem-solving.
Each of my mentors is unique and has his or her strengths, which I look forward to learning from. </p>
<p>Finally, I hope that — indeed, I would be honoured if — my work could make distributed software more secure.
Delivering a tool to concretely help programmers in their message-passing protocol design and implementations would be very gratifying.</p>
<h1 id="questions-for-pauls-nomadic-labs-mentors-zaynah-dargaye-thomas-letan-and-yann-regis-gianas"><a class="toclink" href="#questions-for-pauls-nomadic-labs-mentors-zaynah-dargaye-thomas-letan-and-yann-regis-gianas">Questions for Paul’s Nomadic Labs mentors Zaynah Dargaye, Thomas Letan and Yann Régis-Gianas</a></h1>
<h2 id="what-is-your-input-to-pauls-thesis"><a class="toclink" href="#what-is-your-input-to-pauls-thesis">What is your input to Paul’s thesis?</a></h2>
<p>A thesis is an initiation and apprenticeship into top-level research: we will help Paul to learn how to identify an open problem, understand the state of the art, and study how to build on that state of the art to find a solution.
There are so many skills involved in doing this, and there is so much knowledge required to apply these skills: domain-specific mathematical knowledge, research skills to evaluate which research paths are likely to bear fruit and are worth pursuing, and how to kindle that spark of creative insight which can sometimes make all the difference to finding the solution to a stubborn problem. </p>
<p>Furthermore, this field is an industrial field so solutions also have to be practical.
Paul will have to learn (and we will have to teach him) what the practical requirements and priorities currently are, and how to translate theoretical insights into solutions that are applicable to the real and pressing challenges of designing industry-grade distributed systems.</p>
<h2 id="why-is-this-thesis-important"><a class="toclink" href="#why-is-this-thesis-important">Why is this thesis important?</a></h2>
<p>In early 2020 Paul brought up a concern he encountered in ensuring the correct implementation of a component in the Tezos codebase: how to specify and validate the correct implementation of a message-passing protocol.
Curious, he studied some solutions in existing formalisms, but felt unsatisfied.</p>
<p>The problem that Paul highlighted is widespread in modern systems, whether or not they are distributed: it is not enough that each component of a software system should satisfy its specification locally; they also have to talk together correctly according to some agreed protocol.
In other words, we need a language to specify and reason about how components interact, and what it means for these interactions to be correct (bug-free).</p>
<p>Paul’s thesis introduces a formal language to describe the execution of message-passing protocols rigorously and unambiguously.
From that description, one can generate a reference implementation for each party in a protocol that is <strong>correct-by-construction</strong>: given that the implementation is formally constructed from its specification, it is guaranteed that each component responds correctly to the received messages, and sends the messages predicted by the protocol.
The wonderful thing is that we can deploy this reference implementation in production, and also use it to validate other implementations — e.g. optimised ones that are faster or have better memory-consumption.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:shell">
<p><em>Tezos is split into two parts: the user-facing (and user-upgradable) <a href="https://tezos.gitlab.io/shell/the_big_picture.html#the-embedded-economic-protocols"><strong>Economic Protocol</strong></a>, which is what you see and interact with when you run Tezos — and <a href="https://tezos.gitlab.io/shell/the_big_picture.html#the-shell">the hardware-facing <strong>Shell</strong></a>, which is a lower-level abstraction layer which takes care of e.g. messaging and communication. Think: Economic Protocol = <span class="caps">OS</span> and Shell = Kernel.</em> If Tezos were a car, then the Shell team would be a bunch of mechanics covered in grease, making sure all the bolts are tight and no users get hot oil squirted in their faces (no offense, Paul). -ed <a class="footnote-backref" href="#fnref:shell" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:desired">
<p>Examples range from generic correctness properties, such as no <em>deadlock</em>, when two or more processes get stuck waiting on action from one another (like two super-polite people waiting, forever, for the other one to go through the door first!); all the way to very specific technical correctness properties, such as that the trustworthiness of a certain percentage of the peers advertised to bootstrapping nodes by an advertiser node, is correct with respect to its pool of connections. <a class="footnote-backref" href="#fnref:desired" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:chor">
<p>For choreographies see <a href="https://www.fabriziomontesi.com/files/choreographic_programming.pdf">Fabrizio Montesi’s 2013 PhD thesis <em>“Choreographic Programming”</em></a> (<a href="https://web.archive.org/web/20210622080331/https://www.fabriziomontesi.com/files/choreographic_programming.pdf">permalink</a>) and <a href="https://arxiv.org/abs/1510.03271"><em>“A Core Model for Choreographic Programming”</em> by Luís Cruz-Filipe and Fabrizio Montesi (2016)</a> (see in particular page 1, final paragraph). For endpoint projection, see <a href="https://www.fabriziomontesi.com/files/cm13.pdf"><em>“Deadlock-freedom-by-design: multiparty asynchronous global programming”</em> by Marco Carbone and Fabrizio Montesi (2013)</a> (<a href="https://web.archive.org/web/20210417184935/https://www.fabriziomontesi.com/files/cm13.pdf">permalink</a>). <a class="footnote-backref" href="#fnref:chor" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:network-split">
<p>For example, if communications in a distributed system become incoherent then the network might <em>split</em>, meaning that a subset of the system’s nodes become isolated and evolve independently for some time, and once this has happened it may not be trivial to ‘glue’ the system back together again. <a class="footnote-backref" href="#fnref:network-split" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:cobol">
<p><a href="https://stackoverflow.blog/2020/04/20/brush-up-your-cobol-why-is-a-60-year-old-language-suddenly-in-demand/">This article on the modern need for <span class="caps">COBOL</span></a> (<a href="https://web.archive.org/web/20210607162602/https://stackoverflow.blog/2020/04/20/brush-up-your-cobol-why-is-a-60-year-old-language-suddenly-in-demand/">permalink</a>) is also a good read. <span class="caps">COBOL</span> is in a sense about as far from modern OCaml in the evolution of industrial programming languages as it is possible to get. <a class="footnote-backref" href="#fnref:cobol" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
</ol>
</div>Granada comparisons bug2021-07-08T09:00:00+02:002021-07-08T09:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-07-08:/granada-comparisons-bug.html<p>As a major development center within the Tezos ecosystem, Nomadic Labs routinely performs ongoing reviews and analysis of the Tezos protocol code. In conducting a recent review of the <a href="https://www.tezosagora.org/proposal/PtGRANADsDU8R9daYKAgWnQYAJ64omN1o3KMGVCykShA97vQbvV">Granada proposal</a>, we identified a low-severity bug that occurs in an uncommon pattern in the handling of comparisons, which we would like to raise awareness of so that developers and bakers can be best informed.</p>
<p>The bug was <a href="https://gitlab.com/tezos/tezos/-/merge_requests/2993/diffs?commit_id=bfe2c5c87646193dc98ff5589d5aeb1b2cf9feca">introduced</a> when refactoring the Michelson interpreter. The refactoring dramatically improves gas consumption (typically by 5x or more), but a case was missed out in the Michelson <code>COMPARE</code> function (for Michelson values): when comparing two pairs where the first element of each pair is an option type set to <code>None</code>, <code>COMPARE</code> concludes that the two values are equal when correct behaviour would be to recursively compare the right parts of the pairs. For example:<code>Pair None 3</code> and <code>Pair None 5</code> would be deemed equal by the Granada <code>COMPARE</code> operator, when they should not be because <code>3</code> and <code>5</code> are not equal. The ability to compare option types in Michelson is fairly new and was introduced in Edo. This was further confirmed after we reviewed the contracts deployed on Mainnet, Florencenet, and Granadanet and found that no contracts currently use the pattern that would trigger this bug. Thus, no current contracts would be affected by this error in the Granada proposal.</p>
<p>Please note that future contracts could be affected in the following two scenarios:</p>
<ol>
<li>comparing values that include option types, or</li>
<li>using values that include option types as keys in sets or maps (big maps are unaffected).</li>
</ol>
<p><strong>Neither use case is common</strong>, and even if they do appear, they only affect the contract using it, not the protocol as a whole.</p>
<p>While the Michelson interpreter is <a href="https://research-development.nomadic-labs.com/measuring-test-coverage.html#test-coverage-for-the-michelson-interpreter">well covered</a> by tests and pair and option types are tested, the uncommon combination that triggers this bug was not, so this bug was missed by the test suite.</p>
<p>To better detect such errors in future, we updated our test suite to use property-based testing. <strong>Property-based testing</strong> is an approach that automatically generates random test cases that try to break a function’s desired properties, and is better able to help surface these types of bugs caused by uncommon pattern use. We used property-based testing <a href="https://research-development.nomadic-labs.com/progress-report-on-the-verification-of-liquidity-baking-smart-contracts.html">recently</a> to test properties of the smart contracts used in Liquidity Baking.</p>
<p>If Granada is adopted in the next voting period then:</p>
<ol>
<li>A <a href="https://gitlab.com/nomadic-labs/tezos/-/commit/945c642946ba27349e17cafab6a3783b9a906eba">fix</a> for the comparison bug will be included in the subsequent “H”-named protocol. If adopted we would expect this to activate in October 2021.</li>
<li>Although developers are unlikely to use the patterns that produce incorrect behavior as described above, for safety and convenience purposes we will provide a linting tool to help detect them in code.</li>
</ol>Introducing Mi-Cho-Coq v1.02021-07-02T00:00:00+02:002021-07-02T00:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-07-02:/introducing-mi-cho-coq-v10.html<p>The first public release of the Mi-Cho-Coq framework</p><p>It’s our great pleasure to announce <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/releases/version-1.0">Mi-Cho-Coq version 1.0</a>: the first public release of the free and open-source <strong>Mi-Cho-Coq framework</strong>, a library for verifying the correctness of <a href="https://tezos.gitlab.io/active/michelson.html">Michelson</a> smart contract in <a href="https://coq.inria.fr/">Coq</a> using <a href="https://en.wikipedia.org/wiki/Weakest_precondition_calculus">weakest-precondition calculus</a>.</p>
<p>The <a href="https://gitlab.com/nomadic-labs/mi-cho-coq">Mi-Cho-Coq</a> framework is a <a href="https://coq.inria.fr">Coq</a> library which models all aspects of <a href="https://tezos.gitlab.io/active/michelson.html">the Michelson language</a>: its syntax, its type system, and its semantics.</p>
<p>Although this is the first public release, Mi-Cho-Coq has been in internal use since 2019, including functional verification of some quite large-scale code (some of which is now live):</p>
<ol>
<li>the <a href="https://research-development.nomadic-labs.com/formally-verifying-a-critical-smart-contract.html#verifying-spending-limit-in-mi-cho-coq">spending-limit contract</a>,</li>
<li>several implementations of the <a href="https://research-development.nomadic-labs.com/fa12-approvable-ledger-formal-verification-by-nomadic-labs.html"><span class="caps">FA1</span>.2 token standard</a>, and</li>
<li>several versions of the Dexter decentralized exchange (<a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/merge_requests/100">Dexter v2</a>, <a href="https://research-development.nomadic-labs.com/progress-report-on-the-verification-of-liquidity-baking-smart-contracts.html">Liquidity Baking</a>).</li>
</ol>
<p>Mi-Cho-Coq also features a simple certified Michelson optimizer<sup id="fnref:optimizer"><a class="footnote-ref" href="#fn:optimizer">1</a></sup> and it can also be used as a standalone Michelson type checker<sup id="fnref:typechecker"><a class="footnote-ref" href="#fn:typechecker">2</a></sup></p>
<p>For more details on Mi-Cho-Coq, see <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/blob/master/README.org">the Mi-Cho-Coq <span class="caps">README</span></a>.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:optimizer">
<p>A <em>Michelson optimizer</em> is a tool for turning a Michelson script into a semantically equivalent but more efficient one (i.e. it does the same thing, but quicker). Most compilers targeting Michelson come with Michelson optimizers specialized to that compiler’s output. Mi-Cho-Coq’s Michelson optimizer was initially developed for the <a href="https://albert-lang.io">Albert</a> compiler. For more details about this optimizer, see section 3.3 of <a href="https://link.springer.com/chapter/10.1007/978-3-030-61467-6_5">our article</a> (see also <a href="https://arxiv.org/abs/2106.12973">authors’ pdf</a>). <a class="footnote-backref" href="#fnref:optimizer" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:typechecker">
<p>So a smart-contract programmer can use Mi-Cho-Coq as a lightweight Michelson smart contract type checker, without having to necessarily run <code>tezos-client</code>. <a class="footnote-backref" href="#fnref:typechecker" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>Announcing Octez2021-06-21T09:00:00+02:002021-06-21T09:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-06-21:/announcing-octez.html<p>A veteran implementation of Tezos gets a name: Octez</p><h2 id="rome-the-city-rome-the-ideal"><a class="toclink" href="#rome-the-city-rome-the-ideal">Rome the city, Rome the ideal</a></h2>
<p>There’s a <a href="https://youtu.be/h8MtObh-wDo?t=136">powerful scene in the film the Gladiator</a> where the Emperor Marcus Aurelius explains that Rome is both a physical installation of bricks and mortar, and also it is the <em>idea</em> of Rome — an ecosystem of standards and laws by which the people lived and a city was built.</p>
<p>In other words: Rome is a city, and an ideal.</p>
<p>When you download Tezos, you are actually downloading code that runs on a machine and in so doing embodies <em>the idea of Tezos</em>.</p>
<p>In other words: Tezos is an implementation, and an ideal.</p>
<p>We at Nomadic Labs are proud to have played a role in <a href="#who">the coalition of programmers that wrote</a> a Tezos implementation which is now widely used in the community.
Historically, this was the first complete implementation of Tezos, which was used to activate Tezos Mainnet (the live blockchain) <a href="https://tzstats.com/0">back in 2018</a>.
You can download this implementation from the open source repository <a href="https://gitlab.com/tezos/tezos/">https://gitlab.com/tezos/tezos/</a>, where it is actively maintained today.</p>
<p>However, we were so excited by this at the time that — much like the Romans — we neglected to distinguish linguistically between the ideal, and the implementation of that ideal.
This may be forgivable for world-spanning preindustrial empires, but we modern software developers should be more precise.
So …</p>
<h2 id="the-implementation-is-octez-the-standard-is-tezos"><a class="toclink" href="#the-implementation-is-octez-the-standard-is-tezos">The implementation is <strong>Octez</strong>; the standard is <strong>Tezos</strong></a></h2>
<p>We are happy to announce that the bundle of concrete code files maintained at <a href="https://gitlab.com/tezos/tezos/">https://gitlab.com/tezos/tezos/</a> now has a name: <strong>Octez</strong>. </p>
<p><span class="quo">‘</span>Octez’ is a portmanteau of octopus and Tezos, a pun which draws inspiration from <a href="http://tezos.gitlab.io/shell/the_big_picture.html">this big picture of Tezos</a>.<sup id="fnref:logo"><a class="footnote-ref" href="#fn:logo">1</a></sup></p>
<p><a name="image"></a></p>
<p><img alt="The big picture" src="https://research-development.nomadic-labs.com/images/202106-octez-announcement/octopus.svg"></p>
<p><span class="quo">‘</span>Octez’ is also a portmanteau of OCaml and Tezos (OCaml being the main programming language used in Octez), and a pun on ‘octet’. </p>
<!-- ## Announcing Octez: a name for `tezos/tezos`
Tezos is a decentralized network and, as such, it can have several implementations. One of these implementations is https://gitlab.com/tezos/tezos/. We are happy to announce that this particular implementation now has a name: *Octez*.
-->
<h2 id="octez-in-more-detail"><a class="toclink" href="#octez-in-more-detail">Octez, in more detail</a></h2>
<!-- Octez is a name we have given to an existing implementation; specifically [the set of executables](http://tezos.gitlab.io/introduction/howtouse.html#the-binaries) that you get when you compile the source code at <https://gitlab.com/tezos/tezos/>. -->
<p>Octez is an <a href="https://gitlab.com/tezos/tezos/">implementation</a> of <a href="http://tezos.gitlab.io/introduction/howtouse.html#the-binaries">a suite of Tezos-related software</a>.
It lives in this GitLab repo: <a href="https://gitlab.com/tezos/tezos/">https://gitlab.com/tezos/tezos/</a>.</p>
<p>Octez includes:</p>
<ol>
<li>a Tezos node (which you may know as <code>tezos-node</code>);</li>
<li>a Tezos client for this node (<code>tezos-client</code>);</li>
<li>an implementation of the <a href="http://tezos.gitlab.io/shell/the_big_picture.html#the-economic-protocol-environment-and-compiler">environment for the economic protocol</a>;</li>
<li>daemons (baker, accuser and endorser) for protocols which are active on Mainnet; </li>
<li>a remote signer (<code>tezos-signer</code>); </li>
<li>and further tools, such as an encoder-decoder for Tezos data types (<code>tezos-codec</code>); <code>tezos-protocol-compiler</code>; and <code>tezos-validator</code>.</li>
</ol>
<!-- [mjg no specification of the tezos economic protocol. each protocol has a name https://gitlab.com/tezos/tezos/-/tree/master/src https://gitlab.com/tezos/tezos/-/tree/master/src/proto_010_PtGRANAD]. If you have software with some images, you happen to distribute the software with images.
In those protocol directory is a directory lib_protocol. That has the actual protocol; the rest contains stuff that could be considered part of Octez. Baker, endorser, client are all part of Octez. test is not -->
<!--- a name we have given to an existing implementation; specifically [the suite of executables](http://tezos.gitlab.io/introduction/howtouse.html#the-binaries) that you get when you compile the source code at <https://gitlab.com/tezos/tezos/>. -->
<p>Everything in <a href="#image">the big picture above</a> is Octez, except for </p>
<ul>
<li>the Network (underneath the legs), </li>
<li>the Explorer (bottom right), and</li>
<li>the economic protocol (the green bit with <span class="caps">PROTOCOL</span> written in it) — though Octez is distributed with the economic protocols of Mainnet for convenience.<sup id="fnref:protocol"><a class="footnote-ref" href="#fn:protocol">2</a></sup></li>
</ul>
<h2 id="origin-of-the-implementation"><a class="toclink" href="#origin-of-the-implementation">Origin of the implementation</a></h2>
<p><a name="who"></a>
Octez was created by a coalition of teams including <a href="https://www.nomadic-labs.com">Nomadic Labs</a>, <a href="https://tarides.com">Tarides</a>, <a href="https://tqtezos.com/">Tocqueville Group</a>, <a href="https://obsidian.systems">Obsidian Systems</a>, <a href="https://www.tweag.io">Tweag</a>, and <a href="https://metastate.dev">Metastate</a> (non-exclusive list).</p>
<p>The work is coordinated by the <a href="http://tezos.gitlab.io/developer/merge_team.html">merge team</a> (<a href="https://gitlab.com/tezos/tezos/-/project_members">list of members here</a>), and the name Octez was proposed to and approved by them.
This reflects the decentralized nature of Tezos: Octez is a decentralized implementation, and the name itself was chosen in collaboration.</p>
<h2 id="why-choose-a-name-now"><a class="toclink" href="#why-choose-a-name-now">Why Choose a Name Now?</a></h2>
<!-- The Octez implementation at <https://gitlab.com/tezos/tezos/> is not "Tezos". Tezos is not an implementation, it is a set of protocols (the P2P protocol, the economic protocol, …). The code at https://gitlab.com/tezos/tezos/ is one implementation of these protocols. -->
<p>When we release a new version of <a href="https://gitlab.com/tezos/tezos/">https://gitlab.com/tezos/tezos/</a> we may say something like</p>
<blockquote>
<p>version 9.2 has just been released </p>
</blockquote>
<p>This invites the question: Version 9.2 of what?
The answer is </p>
<blockquote>
<p>Version 9.2 of the Tezos implementation that lives at <a href="https://gitlab.com/tezos/tezos/">https://gitlab.com/tezos/tezos/</a>, </p>
</blockquote>
<p>which is a bit of a mouthful.
So henceforth we can write</p>
<blockquote>
<p>Version 9.2 of <a href="https://gitlab.com/tezos/tezos/">Octez</a>, </p>
</blockquote>
<p>with a clear conscience, complete precision — and our <a href="https://en.wikipedia.org/wiki/Search_engine_optimization"><span class="caps">SEO</span></a> officer would feel better about it too, if we had one.</p>
<p>Thanks for reading … and keep an eye out for a future post titled “Releasing version 10.0 of Octez”!</p>
<!-- https://gitlab.com/tezos/tezos/-/tree/v9.2 You can ignore tag 9.2.0. It is actually exactly the same tag as v9.2 (same commit hash). Tag 9.2.0 only exists because of a technical limitation which is irrelevant for users :) -->
<div class="footnote">
<hr>
<ol>
<li id="fn:logo">
<p>This same picture inspired the <a href="https://web.archive.org/web/20210428215344im_/https://nomadic-labs.com/images/logo.png">Nomadic Labs logo</a>. <a class="footnote-backref" href="#fnref:logo" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:protocol">
<p>See for example the <a href="https://gitlab.com/tezos/tezos/-/tree/master/src/proto_009_PsFLoren/lib_protocol">files for the Florence economic protocol</a>, which is what Mainnet is running at time of writing. <a class="footnote-backref" href="#fnref:protocol" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>FA1.2 Approvable Ledger, formal verification by Nomadic Labs2021-06-15T16:00:00+02:002021-06-15T16:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-06-15:/fa12-approvable-ledger-formal-verification-by-nomadic-labs.html<p>We describe the formal verification in Coq of three <span class="caps">FA1</span>.2-compliant smart contracts.</p><h2 id="introduction"><a class="toclink" href="#introduction">Introduction</a></h2>
<h3 id="overview"><a class="toclink" href="#overview">Overview</a></h3>
<p>We’d like to describe a recent verification effort at Nomadic Labs, namely: </p>
<ul>
<li>A formalisation in Coq of the <span class="caps">FA1</span>.2 standard; </li>
<li>verification in Coq of the formal correctness of three <span class="caps">FA1</span>.2 smart contracts with respect to this formalisation; and</li>
<li>a description of what we learned from our effort, and of the changes and updates made to the standard and the implementations, following our checks.</li>
</ul>
<p>Background, precise links, and definitions of technical terms<sup id="fnref:swallow"><a class="footnote-ref" href="#fn:swallow">1</a></sup> follow below.
Here are the Coq files:</p>
<ul>
<li><a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/merge_requests/97">The GitLab Merge Request of the verification effort</a> (also the <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/blob/kristina-fa12-verification-rebase/src/contracts_coq">corresponding GitLab branch</a>).</li>
<li>A <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq&revision=d116674d67eca4b4b15d7c08efd878a8c8aa864b">snapshot of this GitLab branch on Software Heritage</a>. All relevant code links in this blog post will point here. </li>
<li><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v">The <span class="caps">FA1</span>.2 interface</a></li>
<li><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_spec.v">The <span class="caps">FA1</span>.2 specification</a></li>
<li><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_camlcase.v">Per-contract proof file for the camlCase implementation</a> </li>
<li><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_edukera.v">Per-contract proof file for the Edukera implementation</a> </li>
<li><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_dexter.v">Per-contract proof file for the Dexter 2 implementation</a> </li>
</ul>
<h3 id="background-ledgers-as-smart-contracts"><a class="toclink" href="#background-ledgers-as-smart-contracts">Background: ledgers as smart contracts</a></h3>
<p>Most blockchains have a <strong>native token</strong>.
Tezos has <em>tez</em>;
Bitcoin has <em>bitcoin</em>; and
Ethereum has <em>ether</em>.
These tokens can be used to <a href="https://en.wikipedia.org/wiki/Store_of_value">store value</a>, just like fiat currency.<sup id="fnref:it"><a class="footnote-ref" href="#fn:it">2</a></sup>
Tezos has no other primitive tokens — but it has smart contracts, which gives flexibility to <em>create</em> further tokens and more generally to store value in novel ways.</p>
<p>In fact, Tezos can express any token you can imagine:
just code a smart contract to keep a ledger of who owns your token and how much, and invoke your contract on the Tezos blockchain.
<em>Et voilà</em>: you have a new token on the Tezos blockchain, whose ownership is represented as a ledger state in your smart contract, along with whatever other functionalities you imagined and implemented — e.g. to convert between ledger entries and <span class="caps">USD</span>, like <a href="usdtz.com">USDtz</a>. </p>
<p>However, to do this in a scalable and uniform manner, we need interoperability standards to which such smart contracts can adhere.
Enter the <span class="caps">FA1</span>.2 standard:</p>
<h2 id="the-fa12-standard"><a class="toclink" href="#the-fa12-standard">The <span class="caps">FA1</span>.2 standard</a></h2>
<h3 id="what-fa12-is"><a class="toclink" href="#what-fa12-is">What <span class="caps">FA1</span>.2 is</a></h3>
<p><a href="https://tzip.tezosagora.org/proposal/tzip-7/">The <span class="caps">FA1</span>.2 standard</a> is a standard for smart contracts implementing token ledgers on the Tezos blockchain.<sup id="fnref:erc20"><a class="footnote-ref" href="#fn:erc20">3</a></sup>
The <span class="caps">FA1</span>.2 standard is a plain English document (i.e. not code, but a description of what code should do), officially registered as <a href="https://tzip.tezosagora.org/proposal/tzip-7/"><strong>Tezos Improvement Proposal</strong> (<strong><span class="caps">TZIP</span></strong>) number 7</a>.<sup id="fnref:fa12tzip"><a class="footnote-ref" href="#fn:fa12tzip">4</a></sup> </p>
<p>An <span class="caps">FA1</span>.2-compliant smart contract implements a ledger that maps <strong>owners</strong> to account balances (just like a bank ledger maps account owners to account balances), and offers entrypoints (<a href="#entrypoints">described below</a>) to get balances, approve transfers,<sup id="fnref:approve-transfer"><a class="footnote-ref" href="#fn:approve-transfer">5</a></sup> and so forth, satisfying certain properties as stated in the standard.</p>
<p>For the rest of this blog post, we may write <strong><span class="caps">FA1</span>.2 contract</strong> as shorthand for “a smart contract that satisfies the <span class="caps">FA1</span>.2 standard”.</p>
<h3 id="entrypoints-of-an-fa12-smart-contract"><a class="toclink" href="#entrypoints-of-an-fa12-smart-contract">Entrypoints of an <span class="caps">FA1</span>.2 smart contract</a></h3>
<p><a name="entrypoints"></a></p>
<p>An <span class="caps">FA1</span>.2-compliant smart contract has (at least) the following five entrypoints:</p>
<ol>
<li><code>%transfer</code> expects a <strong>from</strong> account, a <strong>to</strong> account, and an <strong>amount</strong> of tokens to be transferred, and updates the ledger accordingly.</li>
<li><code>%approve</code> expects an <strong>owner</strong>, a <strong>spender</strong>, and a <strong>new allowance</strong> for the spender, and updates the approvals accordingly.</li>
<li><code>%getBalance</code> expects an <strong>owner</strong> and returns (via a <strong>callback</strong>) the owner’s balance. </li>
<li><code>%getAllowance</code> expects an <strong>owner</strong>, a <strong>spender</strong>, and returns (via a <strong>callback</strong>) the approved allowance for the spender.</li>
<li><code>%getTotalSupply</code> returns (again via a <strong>callback</strong>) the total sum of all balances in the ledger.</li>
</ol>
<p>The list above is non-exclusive: an <span class="caps">FA1</span>.2 contract can include other entrypoints — <em>e.g.</em> for burning and minting tokens, or managing an administrator — but it’s the entrypoints listed above that gives a smart contract the property of being “an <span class="caps">FA1</span>.2 smart contract”. </p>
<p><a name="view"></a></p>
<p>We will call the last three entrypoints — <code>%getBalance</code>, <code>%getAllowance</code>, and <code>%getTotalSupply</code> — <a href="https://gitlab.com/tzip/tzip/-/blob/master/proposals/tzip-4/tzip-4.md#view-entrypoints"><strong>view</strong> entrypoints</a>; in <span class="caps">OOP</span> we might call them <em>getter</em> methods.
The view entrypoints will become particularly interesting <a href="#problems">later on in this post</a>, because thanks to our verification work we discovered an ambiguity in their specification.</p>
<h3 id="examples-of-fa12-contracts"><a class="toclink" href="#examples-of-fa12-contracts">Examples of <span class="caps">FA1</span>.2 contracts</a></h3>
<p>Examples of live <span class="caps">FA1</span>.2 contracts include:</p>
<ol>
<li><a href="https://ethtz.io">ETHtz, a token wrapping Ether (<span class="caps">ETH</span>)</a> (<a href="https://better-call.dev/dapps/ethtz">dapp</a>) </li>
<li>The <a href="https://usdtz.com/"><span class="caps">USD</span>-pegged stablecoin USDtz</a> (<a href="https://better-call.dev/dapps/usdtz">dapp</a>)</li>
<li><a href="https://tzbtc.io/">Wrapped Bitcoin tzBTC</a> (<a href="https://better-call.dev/dapps/tzbtc">dapp</a>)</li>
<li>A <em>liquidity ledger</em> that is part of the <a href="https://dexter.exchange/">Dexter 2</a> system.</li>
</ol>
<p><a name="dexter2"></a>
<em>A word on Dexters 1 and 2:</em>
<a href="https://dexter.exchange/">Dexter</a> is a smart contract to enable trade between tez — the Tezos blockchain’s native token — and any <span class="caps">FA1</span>.2 token.
In the current, live version of Dexter (call it Dexter 1 in this blog post<sup id="fnref:second"><a class="footnote-ref" href="#fn:second">6</a></sup>) the liquidity ledger is integrated in the main contract in a manner that does not conform to <span class="caps">FA1</span>.2.
Dexter 2 is a forthcoming replacement of Dexter 1 which includes various improvements, including a modular internal architecture which includes a distinct, <span class="caps">FA1</span>.2-compliant, liquidity ledger.
<strong>Dexter 2’s liquidity ledger</strong> is the contract that has been verified in this work.</p>
<p>Furthermore, two <span class="caps">FA1</span>.2 contracts exist that are not themselves live on the blockchain but which exist as code made freely available and open-source for anyone to build on and deploy:</p>
<ol>
<li><a href="https://gitlab.com/camlcase-dev/fa1.2">an <span class="caps">FA1</span>.2 contract by camlCase written in Morley</a>, and</li>
<li><a href="https://github.com/edukera/archetype-lang/blob/master/contracts/fa12.arl">an <span class="caps">FA1</span>.2 contract by Edukera written in Archetype</a> (see also <a href="https://docs.archetype-lang.org/contract-library/tokens/fa-1.2">pretty-printed code</a>). </li>
</ol>
<p>So several <span class="caps">FA1</span>.2-compliant ledgers exist for the Tezos blockchain: </p>
<ul>
<li>ledgers in ETHtz, USDtz, and tzBTC; </li>
<li>an internal ledger used by the Dexter 2 smart contract; and </li>
<li>a pair of general-purpose ledger implementations, one by camlCase and one by Edukera.</li>
</ul>
<p>This makes sense, because keeping a ledger is a basic functionality.</p>
<h3 id="two-important-questions"><a class="toclink" href="#two-important-questions">Two important questions</a></h3>
<p>Two questions now arise:</p>
<ol>
<li>
<p><strong>Are the ledgers above correctly implemented: does the implementation comply with the <span class="caps">FA1</span>.2 standard?</strong> </p>
<p>The technical term for this is <strong>(formal) correctness</strong>. Just because an implementation <strong>says</strong> </p>
<blockquote>
<p><em><span class="dquo">“</span>I am a smart contract implementing an <span class="caps">FA1</span>.2-compliant token ledger”</em> </p>
</blockquote>
<p>does not mean it <strong>is</strong> one. The implementation might contain an error.</p>
</li>
<li>
<p><strong>Is the <span class="caps">FA1</span>.2 standard itself correct and unambiguous?</strong></p>
<p>Just because a standard <strong>says</strong> </p>
<blockquote>
<p><em><span class="dquo">“</span>I am a standard”</em> </p>
</blockquote>
<p>does not mean that it <strong>is</strong> one.
The <span class="caps">FA1</span>.2 standard is a plain English document.
This could contain an error, ambiguity, or omission.</p>
</li>
</ol>
<p>We should answer </p>
<ul>
<li>Question 2 (correctness of the standard) before </li>
<li>Question 1 (correctness of an implementation with respect to a formalisation of that standard), </li>
</ul>
<p>since we should check that a standard makes sense before we formalise it and check that an implementation is formally correct with respect to this formalisation!<sup id="fnref:garbage-in"><a class="footnote-ref" href="#fn:garbage-in">7</a></sup></p>
<h2 id="the-fa12-standard-formalised"><a class="toclink" href="#the-fa12-standard-formalised">The <span class="caps">FA1</span>.2 standard formalised</a></h2>
<h3 id="formalising-fa12-standard-as-fa12-interface-fa12-spec"><a class="toclink" href="#formalising-fa12-standard-as-fa12-interface-fa12-spec">Formalising “<span class="caps">FA1</span>.2 standard” as “<span class="caps">FA1</span>.2 interface” + “<span class="caps">FA1</span>.2 spec”</a></h3>
<p>To answer Question 2 above, we at Nomadic Labs wrote some code in the <a href="https://coq.inria.fr/">Coq proof assistant</a>, as follows:</p>
<ol>
<li>
<p><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v">The <strong><span class="caps">FA1</span>.2 interface</strong></a>.</p>
<p>This asserts internal (intensional) structure which the smart contract must support, along with axioms on its behaviour. </p>
<p>An <span class="caps">FA1</span>.2-compliant smart contract may support other internal structure too.
And why is a file regarding intensional behaviour called an <em>interface</em>?
Because it provides an abstract interface to the implementation’s storage and parameter types; see <a href="#anote">a note below</a>.</p>
</li>
<li>
<p><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_spec.v">The <strong><span class="caps">FA1</span>.2 specification</strong></a>.</p>
<p>This specifies how <span class="caps">FA1</span>.2 entrypoints interact with the internal structure of the <span class="caps">FA1</span>.2 interface.
An <span class="caps">FA1</span>.2-compliant smart contract may contain more entrypoints than those specified in the <span class="caps">FA1</span>.2 standard.</p>
</li>
<li>
<p>There is also an <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_verification.v"><strong><span class="caps">FA12</span> verification</strong></a> file, which collects lemmas proved purely from the interface and specification files.
More on this <a href="#verification_file">later</a>. </p>
</li>
</ol>
<p>So: we translated the English <span class="caps">FA1</span>.2 standard into a pair of Coq file and some corollaries — the first concerning intensional structure, the second concerning extensional behaviour, and the third collecting their consequences when put together — and we can write the following informal equation:</p>
<div class="math">$$
\text{FA1.2 standard} \quad\stackrel{\text{COQ}}\Longrightarrow\quad \text{FA1.2 interface}\ \ +\ \ \text{FA1.2 specification} .
$$</div>
<p>We now discuss the components on the right-hand side of this equation, in more detail.</p>
<p>We start with the <em><span class="caps">FA1</span>.2 interface</em>, which is as discussed a collection of Coq module types that does two things: provide an abstract interface of an <span class="caps">FA1</span>.2 implementation’s storage and parameter types; and impose axiomatic requirements on its behaviour.</p>
<p>We discuss each in turn:</p>
<p><a name="getters-and-setters"></a></p>
<h3 id="the-fa12-interface-functions"><a class="toclink" href="#the-fa12-interface-functions">The <span class="caps">FA1</span>.2 interface (functions)</a></h3>
<p>The <span class="caps">FA1</span>.2 interface posits the following getters and setters over an <span class="caps">FA1</span>.2 implementation’s storage:</p>
<ol>
<li><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v#L115">a function <code>getBalance</code></a>
to return an owner’s balance, or return zero if the owner does not have a balance in the ledger
(<code>getBalance</code> is a totalising wrapper around an <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v#L92">underlying <em>partial</em> function <code>getBalanceOpt</code></a>);<sup id="fnref:annot"><a class="footnote-ref" href="#fn:annot">8</a></sup></li>
<li><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v#L121">a function <code>getAllowance</code></a> to return an owner’s allowance for a spender, or zero if the spender is not recorded as having an allowance with the owner (again, this is a wrapper for an underlying <code>getAllowanceOpt</code>); </li>
<li><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v#L95">a function <code>setBalance</code></a> and
<a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v#L101">a function <code>setAllowance</code></a> to modify the corresponding data; and </li>
<li><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v#L104">a function <code>getTotalSupply</code></a> to obtain the total sum of all tokens in the ledger.</li>
</ol>
<p>Here that is again as Coq code:</p>
<div class="highlight"><pre><span></span><code><span class="n">getBalance</span> <span class="o">:</span> <span class="n">data</span> <span class="n">storage_ty</span> <span class="o">-></span> <span class="n">data</span> <span class="n">address</span> <span class="o">-></span> <span class="n">data</span> <span class="n">nat</span> <span class="c">(** is wrapper for ... **)</span>
<span class="n">getBalanceOpt</span> <span class="o">:</span> <span class="n">data</span> <span class="n">storage_ty</span> <span class="o">-></span> <span class="n">data</span> <span class="n">address</span> <span class="o">-></span> <span class="n">data</span> <span class="o">(</span><span class="n">option</span> <span class="n">nat</span><span class="o">)</span>
<span class="n">getAllowance</span> <span class="o">:</span> <span class="n">data</span> <span class="n">storage_ty</span> <span class="o">-></span> <span class="n">data</span> <span class="n">address</span> <span class="o">-></span> <span class="n">data</span> <span class="n">address</span> <span class="o">-></span> <span class="n">data</span> <span class="n">nat</span>
<span class="n">getAllowanceOpt</span> <span class="o">:</span> <span class="n">data</span> <span class="n">storage_ty</span> <span class="o">-></span> <span class="n">data</span> <span class="n">address</span> <span class="o">-></span> <span class="n">data</span> <span class="n">address</span> <span class="o">-></span> <span class="n">data</span> <span class="o">(</span><span class="n">option</span> <span class="n">nat</span><span class="o">)</span>
<span class="n">setBalance</span> <span class="o">:</span> <span class="n">data</span> <span class="n">storage_ty</span> <span class="o">-></span> <span class="n">data</span> <span class="n">address</span> <span class="o">-></span> <span class="n">data</span> <span class="n">nat</span> <span class="o">-></span> <span class="n">data</span> <span class="n">storage_ty</span>
<span class="n">setAllowance</span> <span class="o">:</span> <span class="n">data</span> <span class="n">storage_ty</span> <span class="o">-></span> <span class="n">data</span> <span class="n">address</span> <span class="o">-></span> <span class="n">data</span> <span class="n">address</span> <span class="o">-></span>
<span class="n">data</span> <span class="n">nat</span> <span class="o">-></span> <span class="n">data</span> <span class="n">storage_ty</span>
<span class="n">getTotalSupply</span> <span class="o">:</span> <span class="n">data</span> <span class="n">storage_ty</span> <span class="o">-></span> <span class="n">data</span> <span class="n">nat</span>
</code></pre></div>
<p>For experts: <code>data</code> above is a wrapper that turns a Michelson type into a Coq type.
To be more precise, it is a Coq function that translates an underlying <a href="https://arxiv.org/abs/1909.08671">Mi-Cho-Coq representation of Michelson types</a>, to <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/michocoq/semantics.v#L281">a type <code>data</code> in Coq</a>.</p>
<p><a name="anote"></a></p>
<p>Note that </p>
<ul>
<li>the functions <code>getBalance</code>, <code>getAllowance</code>, <code>setBalance</code>, <code>setAllowance</code>, and <code>getTotalSupply</code> might be present in the smart contract’s source code as explicit internal functions — e.g. one way to write an <span class="caps">FA1</span>.2-compliant smart contract is to use a smart contracts language that is a functional language and allows us to actually write these functions — but also</li>
<li>these functions might be implicitly definable from the basic datatypes of the implementation, but not explicitly represented as such — e.g. perhaps the smart contract is written in a low-level language that simply does not have functions.</li>
</ul>
<p>Part of the power of the interface is that it makes this distinction not matter so much: in the rest of the verification we can operate <em>as if</em> <code>getBalance</code> exists, even if it’s only implicit.</p>
<h3 id="the-fa12-interface-axioms"><a class="toclink" href="#the-fa12-interface-axioms">The <span class="caps">FA1</span>.2 interface (axioms)</a></h3>
<p>The <span class="caps">FA1</span>.2 interface also imposes axioms on the <a href="#getters-and-setters">getters and setters above</a>.
For example:</p>
<h4 id="example-axiom-1"><a class="toclink" href="#example-axiom-1">Example axiom 1</a></h4>
<p><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v#L177"><code>setBalance</code> to <code>amount</code>, followed by <code>getBalance</code>, should return the <code>amount</code></a>.
In Coq:</p>
<div class="highlight"><pre><span></span><code><span class="kn">Parameter</span> <span class="n">getBalance_setBalance_eq</span> <span class="o">:</span> <span class="k">forall</span> <span class="n">sto</span> <span class="n">owner</span> <span class="n">amount</span><span class="o">,</span>
<span class="n">getBalance</span> <span class="o">(</span><span class="n">setBalance</span> <span class="n">sto</span> <span class="n">owner</span> <span class="n">amount</span><span class="o">)</span> <span class="n">owner</span> <span class="o">=</span> <span class="n">amount</span><span class="o">.</span>
</code></pre></div>
<h4 id="example-axiom-2"><a class="toclink" href="#example-axiom-2">Example axiom 2</a></h4>
<p><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v#L180">Setting a balance of one owner must leave the balances of other owners unchanged</a>:</p>
<div class="highlight"><pre><span></span><code><span class="kn">Parameter</span> <span class="n">getBalance_setBalance_neq</span> <span class="o">:</span> <span class="k">forall</span> <span class="n">sto</span> <span class="n">owner</span> <span class="n">owner'</span> <span class="n">amount</span><span class="o">,</span>
<span class="n">owner</span> <span class="o"><></span> <span class="n">owner'</span> <span class="o">-></span>
<span class="n">getBalance</span> <span class="o">(</span><span class="n">setBalance</span> <span class="n">sto</span> <span class="n">owner</span> <span class="n">amount</span><span class="o">)</span> <span class="n">owner'</span> <span class="o">=</span>
<span class="n">getBalance</span> <span class="n">sto</span> <span class="n">owner'</span><span class="o">.</span>
</code></pre></div>
<p><em>For experts:</em> Example axioms 1 and 2 state that the ledger is an <a href="https://en.wikipedia.org/wiki/Array_data_type#Abstract_arrays">abstract array</a>. </p>
<h4 id="example-axiom-3"><a class="toclink" href="#example-axiom-3">Example axiom 3</a></h4>
<p><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v#L185">Setting an allowance for an owner and a spender must have no impact on anyone’s balance values</a>:</p>
<div class="highlight"><pre><span></span><code><span class="kn">Parameter</span> <span class="n">getAllowance_setAllowance_eq</span> <span class="o">:</span> <span class="k">forall</span> <span class="n">sto</span> <span class="n">owner</span> <span class="n">spender</span> <span class="n">amount</span><span class="o">,</span>
<span class="n">getAllowance</span> <span class="o">(</span><span class="n">setAllowance</span> <span class="n">sto</span> <span class="n">owner</span> <span class="n">spender</span> <span class="n">amount</span><span class="o">)</span> <span class="n">owner</span> <span class="n">spender</span> <span class="o">=</span>
<span class="n">amount</span><span class="o">.</span>
</code></pre></div>
<h4 id="summary"><a class="toclink" href="#summary">Summary</a></h4>
<p>So: the <span class="caps">FA1</span>.2 interface is Coq code, not English.
Like the <span class="caps">FA1</span>.2 standard from which it derives, the <span class="caps">FA1</span>.2 interface abstracts away from smart contract implementation details to allow us to carry out aspects of the verification once and for all, in an implementation-independent manner.</p>
<p>With this in hand, we can conveniently state the <span class="caps">FA1</span>.2 specification:</p>
<h3 id="the-fa12-specification"><a class="toclink" href="#the-fa12-specification">The <span class="caps">FA1</span>.2 specification</a></h3>
<p>The <em><span class="caps">FA1</span>.2 specification</em> specifies how smart contract entrypoints should invoke functions specified in the <span class="caps">FA1</span>.2 interface — in other words, it specifies the effect of executing <span class="caps">FA1</span>.2 entrypoints in terms of the abstract view of the contract given by the interface.</p>
<p>Excerpts from <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_spec.v#L95">here</a> and <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_spec.v#L126">here</a>:</p>
<div class="highlight"><pre><span></span><code><span class="c">(** Entry point: ep_getBalance *)</span>
<span class="kn">Definition</span> <span class="n">ep_getBalance</span>
<span class="o">(</span><span class="n">p</span> <span class="o">:</span> <span class="n">data</span> <span class="n">parameter_ep_getBalance_ty</span><span class="o">)</span>
<span class="o">(</span><span class="n">sto</span> <span class="o">:</span> <span class="n">data</span> <span class="n">storage_ty</span><span class="o">)</span>
<span class="o">(</span><span class="n">ret_ops</span> <span class="o">:</span> <span class="n">data</span> <span class="o">(</span><span class="n">list</span> <span class="n">operation</span><span class="o">))</span>
<span class="o">(</span><span class="n">ret_sto</span> <span class="o">:</span> <span class="n">data</span> <span class="n">storage_ty</span><span class="o">)</span> <span class="o">:=</span>
<span class="k">let</span> <span class="k">'</span><span class="o">(</span><span class="n">owner</span><span class="o">,</span> <span class="n">callback</span><span class="o">)</span> <span class="o">:=</span> <span class="n">p</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">balance</span> <span class="o">:=</span> <span class="n">getBalance</span> <span class="n">sto</span> <span class="n">owner</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">op</span> <span class="o">:=</span> <span class="n">transfer_tokens</span> <span class="n">env</span> <span class="n">nat</span> <span class="n">balance</span> <span class="o">(</span><span class="n">amount</span> <span class="n">env</span><span class="o">)</span> <span class="n">callback</span> <span class="k">in</span>
<span class="n">ret_sto</span> <span class="o">=</span> <span class="n">sto</span> <span class="o">/\</span> <span class="n">ret_ops</span> <span class="o">=</span> <span class="o">[</span><span class="n">op</span><span class="o">].</span>
</code></pre></div>
<p>Here, we specify in <code>ep_getBalance</code> that the behavior of the <code>%getBalance</code> entrypoint is equivalent to retrieving the balance of the <code>owner</code> (using the <code>getBalance</code> getter provided by the interface) and sending it to <code>callback</code>, through the emission of a transfer operation.</p>
<p>The specification looks like code but it’s not: it specifies required entrypoints and their parameters and behaviour, but treats them as opaque structures, the internal composition of which we do not examine.</p>
<!-- Let's reprise an overview of the moving parts in play here.
We have:
1. **Concrete entrypoints** of a specific FA1.2 implementation. This is code.
1. **Functions in the FA1.2 interface.**
These specify how concrete entrypoints in the implementation correspond to abstract entrypoints in the FA1.2 specification.
1. **Functions in an FA1.2 specification.** These correspond to abstract entrypoints, on which we impose the restrictions required by the FA1.2 specification.
-->
<h2 id="three-fa12-smart-contract-implementions-verified"><a class="toclink" href="#three-fa12-smart-contract-implementions-verified">Three <span class="caps">FA1</span>.2 smart contract implementions verified</a></h2>
<p>Having written the <span class="caps">FA1</span>.2 interface and specification just discussed,
we verified formal correctness with respect to them, of three <span class="caps">FA1</span>.2 smart contract implementations: </p>
<p><a name="three_implementations"></a> </p>
<ol>
<li><a href="https://gitlab.com/camlcase-dev/fa1.2">an <span class="caps">FA1</span>.2 contract by camlCase</a>, </li>
<li><a href="https://github.com/edukera/archetype-lang/blob/master/contracts/fa12.arl">an <span class="caps">FA1</span>.2 contract by Edukera</a> (<a href="https://docs.archetype-lang.org/contract-library/tokens/fa-1.2">pretty-printed code</a>), and </li>
<li>the <a href="https://gitlab.com/dexter2tz/dexter2tz/">Dexter 2 <span class="caps">FA1</span>.2 implementation</a>. </li>
</ol>
<p><a href="https://www.edukera.com/">Edukera</a> had already checked their implementation using a different methodology. Both the contract and its specification are written in the Archetype language and converted by the Archetype compiler into a collection of proof obligations. These proof obligations are subsequently solved using the <a href="http://why3.lri.fr">verification platform Why3</a> by means of <a href="https://forum.tezosagora.org/t/a-verified-implementation-of-fa1-2/2264"><span class="caps">SMT</span> solvers</a>.</p>
<p>Nevertheless, we decided to carry out the verification of the Edukera contract in Coq to put our approach to the test: if the two verification efforts yield the same result, namely a formal proof of correctness, we have a basis for comparison; if not, we investigate any discrepancies.</p>
<h3 id="examples-of-implementational-differences"><a class="toclink" href="#examples-of-implementational-differences">Examples of implementational differences</a></h3>
<p>The three implementations maintain internal storage differently:</p>
<h4 id="map-of-approved-balances-ledger-of-balances"><a class="toclink" href="#map-of-approved-balances-ledger-of-balances">Map of approved balances / ledger of balances</a></h4>
<p>In the Edukera and Dexter 2 implementations, the map of approved allowances across owners is stored separately from the ledger of balances.
The Edukera storage is</p>
<div class="highlight"><pre><span></span><code><span class="k">storage</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">big_map</span><span class="w"> </span><span class="k k-Name">%allowance</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="kt">address</span><span class="w"> </span><span class="kt">address</span><span class="p">)</span><span class="w"> </span><span class="kt">nat</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">big_map</span><span class="w"> </span><span class="k k-Name">%ledger</span><span class="w"> </span><span class="kt">address</span><span class="w"> </span><span class="kt">nat</span><span class="p">))</span>
</code></pre></div>
<p>and the Dexter 2 storage is</p>
<div class="highlight"><pre><span></span><code><span class="k">storage</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">big_map</span><span class="w"> </span><span class="k k-Name">%tokens</span><span class="w"> </span><span class="kt">address</span><span class="w"> </span><span class="kt">nat</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">big_map</span><span class="w"> </span><span class="k k-Name">%allowances</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">%owner</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">%spender</span><span class="p">))</span><span class="w"> </span><span class="kt">nat</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">%admin</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">%total_supply</span><span class="p">))))</span>
</code></pre></div>
<p>In contrast, the camlCase implementation maintains one <a href="https://tezos.gitlab.io/michelson-reference/#type-big_map">big map</a> mapping an owner to a pair whose first component is the owner’s balance and the second component is a map of the owner’s approved allowances: </p>
<div class="highlight"><pre><span></span><code><span class="k">storage</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">big_map</span><span class="w"> </span><span class="k k-Name">%accounts</span><span class="w"> </span><span class="kt">address</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">:balance</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">map</span><span class="w"> </span><span class="k k-Name">:approvals</span><span class="w"> </span><span class="kt">address</span>
<span class="w"> </span><span class="kt">nat</span><span class="p">)))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">%fields</span><span class="p">));</span>
</code></pre></div>
<p>(In <a href="https://gitlab.com/camlcase-dev/fa1.2/-/blob/master/morley/fa1.2.tz#L13">the code above</a>, <code>%fields</code> corresponds to the total supply.)</p>
<p>Thus, the Edukera and Dexter 2 contracts admit an account that is absent from the ledger — not even with a zero balance — yet has approved spenders in the map of allowances.</p>
<p>In the camlCase contract this cannot happen as there is no global map of allowances separate from the ledger; instead, each owner maintains its own map of allowances.</p>
<p><a name="additional_entrypoints"></a></p>
<h4 id="additional-entrypoints"><a class="toclink" href="#additional-entrypoints">Additional entrypoints</a></h4>
<p>Contracts providing functionalities beyond the <span class="caps">FA1</span>.2 standard may have additional fields in their storage.
For instance, the Dexter 2 contract has a field called <code>%admin</code> for an administrator account.</p>
<p>Admin is the only account allowed to burn and/or mint tokens in Dexter 2, using an additional <code>%mintOrBurn</code> entrypoint. The Edukera and camlCase implementations have no entrypoints aside from those specified in the <span class="caps">FA1</span>.2 interface.</p>
<h2 id="guarantees-provided-by-formal-correctness"><a class="toclink" href="#guarantees-provided-by-formal-correctness">Guarantees provided by formal correctness</a></h2>
<p>Correctness with respect to the <span class="caps">FA1</span>.2 interface and specification ensures that certain things are guaranteed, regardless of any implementational differences.
For example:</p>
<p><a name="balances-unchanged"></a></p>
<h3 id="sum-of-balances-is-unchanged"><a class="toclink" href="#sum-of-balances-is-unchanged">Sum of balances is unchanged</a></h3>
<p>If an implementation is correct, then regardless of </p>
<ul>
<li>the specific form of storage, or </li>
<li>whether or not there are any additional entrypoints, </li>
</ul>
<p>each of <a href="#entrypoints">the five <span class="caps">FA1</span>.2 entrypoints</a> must leave the total sum of balances unchanged.<sup id="fnref:others"><a class="footnote-ref" href="#fn:others">9</a></sup></p>
<p><a name="verification_file"></a></p>
<p>This is in <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_verification.v">a file <code>fa12_verification</code></a>, which consists of a number of useful lemmas and a <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_verification.v#L189">culminating <code>Theorem sumOfAllBalances_constant</code></a>. </p>
<p>This theorem is checked once-and-for-all, purely from an assumption that the smart contract is <span class="caps">FA1</span>.2-compliant — no need to re-run the proof for each implementation; no need even to look at the implementation code (once we have proved that it is <span class="caps">FA1</span>.2-compliant).</p>
<p><a name="storage"></a></p>
<h3 id="storage-validity"><a class="toclink" href="#storage-validity">Storage Validity</a></h3>
<p>The <span class="caps">FA1</span>.2 standard states (in English) that the <code>%getTotalSupply</code> entrypoint must return the total sum of all tokens in the ledger.<sup id="fnref:does-not"><a class="footnote-ref" href="#fn:does-not">10</a></sup></p>
<p>The <span class="caps">FA1</span>.2 interface posits a function <code>getTotalSupply : data storage_ty -> data nat</code>, and the <span class="caps">FA1</span>.2 specification requires that that the value returned by the <span class="caps">FA1</span>.2 <code>%getTotalSupply</code> entrypoint is equal to the value we get by calling <code>getTotalSupply</code> directly
(in other words: the specification insists that the <code>%getTotalSupply</code> entrypoint is equivalent to calling the <code>getTotalSupply</code> function directly on the storage).</p>
<p><em>However</em> this in itself does not guarantee that the <code>getTotalSupply</code> function actually <em>returns the sum of all tokens</em>.
This must be checked on a per-implementation basis, as we now discuss. </p>
<p>The <span class="caps">FA1</span>.2 interface <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v#L149">defines a <code>storageValid</code> predicate</a> —</p>
<div class="highlight"><pre><span></span><code><span class="kn">Definition</span> <span class="n">storageValid</span> <span class="n">sto</span> <span class="o">:=</span>
<span class="n">getTotalSupply</span> <span class="n">sto</span> <span class="o">=</span> <span class="n">sumOfAllBalances</span> <span class="n">sto</span><span class="o">.</span>
</code></pre></div>
<p>But in an implementation we are unlikely to see this summation directly implemented in code, since it is impractical to recompute the total sum of all balances every time the <code>%getTotalSupply</code> entrypoint is invoked —- depending on the implementation language, the data structures may not even permit this iteration (i.e. not <em>foldable</em>), so that the language abstractions might make such a sum impossible to even express (e.g. <a href="https://tezos.gitlab.io/michelson-reference/#type-big_map"><code>big map</code></a>).</p>
<p>Thus, the total supply is stored as a separate field in the storage and updated as tokens are burned or minted.
The camlCase and Dexter 2 implementations do this.
The Edukera implementation does too but since it provides no options for burning or minting tokens, it <a href="https://github.com/edukera/archetype-lang/blob/master/contracts/fa12.arl#L3">just sets total supply to a large constant number</a>: <code>constant totalsupply : nat = 10_000_000</code>.</p>
<p>Now to prove formal correctness we must prove that <code>storageValid</code> is indeed valid:</p>
<ul>
<li><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_camlcase_verification.v#L313">camlCase</a></li>
<li><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_edukera_verification.v#L341">Edukera</a></li>
<li><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_dexter_verification.v#L395">Dexter 2</a></li>
</ul>
<p>The proofs of these properties are quite long, and this is where some real work has to happen: we have to actually read the implementation and <em>prove</em> in Coq that <code>storageValid</code> — the equality between the value returned by <code>%getTotalSupply</code> and the total sum of all tokens — is maintained as a dynamic invariant preserved by <em>all</em> of the contract’s entrypoints.
<a name="every_entrypoint"></a>
This includes entrypoints that are not required by the <span class="caps">FA1</span>.2 standard, e.g. entrypoints for minting and burning tokens, if any.</p>
<p>In practice, most of the work was already done by the <code>sumOfAllBalances_constant</code> theorem <a href="#balances-unchanged">previously discussed</a>.
Of the <a href="#three_implementations">three contracts we considered</a> only the the Dexter 2 contract has an additional entrypoint.
Yes this is handled using bespoke reasoning, but even so this reasoning mostly just calls generic lemmas from the <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_verification.v"><code>fa12_verification</code> file</a> which we <a href="#verification_file">mentioned above</a>. </p>
<h3 id="all-entrypoints-are-present"><a class="toclink" href="#all-entrypoints-are-present">All entrypoints are present</a></h3>
<p>The <span class="caps">FA1</span>.2 parameter identifies the entrypoint to be invoked and provides the arguments to go along with the specified entrypoint.
The abstract <span class="caps">FA1</span>.2 interface requires an implementation to provide <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v#L89">a method</a> </p>
<div class="highlight"><pre><span></span><code><span class="n">extract_fa12_ep</span> <span class="o">:</span> <span class="n">data</span> <span class="n">parameter_ty</span> <span class="o">-></span> <span class="n">data</span> <span class="o">(</span><span class="n">option</span> <span class="n">fa12_parameter_ty</span><span class="o">)</span>
</code></pre></div>
<p>that maps an entrypoint of the implementation to a corresponding entrypoint in the <span class="caps">FA1</span>.2 standard.
It is partial (the “<code>option</code>” in <code>option fa12_parameter_ty</code>) because the implementation may have additional entrypoints that do not correspond to any <span class="caps">FA1</span>.2 entrypoint.</p>
<p><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v#L162">We then require</a> that each <span class="caps">FA1</span>.2 entrypoint has a corresponding entrypoint in the implementation — and hence that the implementation is indeed an implementation of <span class="caps">FA1</span>.2:</p>
<div class="highlight"><pre><span></span><code><span class="kn">Parameter</span> <span class="n">ep_transfer_required</span> <span class="o">:</span> <span class="k">forall</span> <span class="o">(</span><span class="n">q</span> <span class="o">:</span> <span class="n">data</span> <span class="n">parameter_ep_transfer_ty</span><span class="o">),</span>
<span class="k">exists</span> <span class="n">p</span><span class="o">,</span> <span class="n">extract_fa12_ep</span> <span class="n">p</span> <span class="o">=</span> <span class="n">Some</span> <span class="o">(</span><span class="n">inl</span> <span class="o">(</span><span class="n">inl</span> <span class="n">q</span><span class="o">)).</span>
<span class="kn">Parameter</span> <span class="n">ep_approve_required</span> <span class="o">:</span> <span class="k">forall</span> <span class="o">(</span><span class="n">q</span> <span class="o">:</span> <span class="n">data</span> <span class="n">parameter_ep_approve_ty</span><span class="o">),</span>
<span class="k">exists</span> <span class="n">p</span><span class="o">,</span> <span class="n">extract_fa12_ep</span> <span class="n">p</span> <span class="o">=</span> <span class="n">Some</span> <span class="o">(</span><span class="n">inl</span> <span class="o">(</span><span class="n">inr</span> <span class="n">q</span><span class="o">)).</span>
<span class="kn">Parameter</span> <span class="n">ep_getAllowance_required</span> <span class="o">:</span> <span class="k">forall</span> <span class="o">(</span><span class="n">q</span> <span class="o">:</span> <span class="n">data</span> <span class="n">parameter_ep_getAllowance_ty</span><span class="o">),</span>
<span class="k">exists</span> <span class="n">p</span><span class="o">,</span> <span class="n">extract_fa12_ep</span> <span class="n">p</span> <span class="o">=</span> <span class="n">Some</span> <span class="o">(</span><span class="n">inr</span> <span class="o">(</span><span class="n">inl</span> <span class="n">q</span><span class="o">)).</span>
<span class="kn">Parameter</span> <span class="n">ep_getBalance_required</span> <span class="o">:</span> <span class="k">forall</span> <span class="o">(</span><span class="n">q</span> <span class="o">:</span> <span class="n">data</span> <span class="n">parameter_ep_getBalance_ty</span><span class="o">),</span>
<span class="k">exists</span> <span class="n">p</span><span class="o">,</span> <span class="n">extract_fa12_ep</span> <span class="n">p</span> <span class="o">=</span> <span class="n">Some</span> <span class="o">(</span><span class="n">inr</span> <span class="o">(</span><span class="n">inr</span> <span class="o">(</span><span class="n">inl</span> <span class="n">q</span><span class="o">))).</span>
<span class="kn">Parameter</span> <span class="n">ep_getTotalSupply_required</span> <span class="o">:</span> <span class="k">forall</span> <span class="o">(</span><span class="n">q</span> <span class="o">:</span> <span class="n">data</span> <span class="n">parameter_ep_getTotalSupply_ty</span><span class="o">),</span>
<span class="k">exists</span> <span class="n">p</span><span class="o">,</span> <span class="n">extract_fa12_ep</span> <span class="n">p</span> <span class="o">=</span> <span class="n">Some</span> <span class="o">(</span><span class="n">inr</span> <span class="o">(</span><span class="n">inr</span> <span class="o">(</span><span class="n">inr</span> <span class="n">q</span><span class="o">))).</span>
</code></pre></div>
<h3 id="the-approve-entrypoint-is-correct"><a class="toclink" href="#the-approve-entrypoint-is-correct">The <code>%approve</code> entrypoint is correct</a></h3>
<p>The specification for the <code>%approve</code> entrypoint guarantees that the storage returned by the contract is obtained from the initial storage by calling <em>setAllowance</em> with the <strong>owner</strong>, <strong>spender</strong>, and <strong>amount</strong> arguments accompanying the call to <code>%approve</code>.</p>
<p>Here’s <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_spec.v#L54">the Coq code</a>:</p>
<div class="highlight"><pre><span></span><code><span class="c">(** Entry point: ep_approve *)</span>
<span class="kn">Definition</span> <span class="n">ep_approve</span>
<span class="o">(</span><span class="n">p</span> <span class="o">:</span> <span class="n">data</span> <span class="n">parameter_ep_approve_ty</span><span class="o">)</span>
<span class="o">(</span><span class="n">sto</span> <span class="o">:</span> <span class="n">data</span> <span class="n">storage_ty</span><span class="o">)</span>
<span class="o">(</span><span class="n">ret_ops</span> <span class="o">:</span> <span class="n">data</span> <span class="o">(</span><span class="n">list</span> <span class="n">operation</span><span class="o">))</span>
<span class="o">(</span><span class="n">ret_sto</span> <span class="o">:</span> <span class="n">data</span> <span class="n">storage_ty</span><span class="o">)</span> <span class="o">:=</span>
<span class="k">let</span> <span class="k">'</span><span class="o">(</span><span class="n">spender</span><span class="o">,</span> <span class="n">new_allowance</span><span class="o">)</span> <span class="o">:=</span> <span class="n">p</span> <span class="k">in</span>
<span class="o">(</span><span class="n">sender</span> <span class="o">=</span> <span class="n">spender</span> <span class="o">/\</span> <span class="n">ret_sto</span> <span class="o">=</span> <span class="n">sto</span><span class="o">)</span> <span class="o">\/</span>
<span class="k">let</span> <span class="n">current_allowance</span> <span class="o">:=</span> <span class="n">getAllowance</span> <span class="n">sto</span> <span class="n">sender</span> <span class="n">spender</span> <span class="k">in</span>
<span class="o">(</span><span class="n">current_allowance</span> <span class="o">=</span> <span class="mi">0</span><span class="o">%</span><span class="n">N</span> <span class="o">\/</span> <span class="n">new_allowance</span> <span class="o">=</span> <span class="mi">0</span><span class="o">%</span><span class="n">N</span><span class="o">)</span> <span class="o">/\</span>
<span class="n">ret_sto</span> <span class="o">=</span> <span class="n">setAllowance</span> <span class="n">sto</span> <span class="n">sender</span> <span class="n">spender</span> <span class="n">new_allowance</span><span class="o">.</span>
</code></pre></div>
<h3 id="underspecification"><a class="toclink" href="#underspecification">Underspecification</a></h3>
<p>The <span class="caps">FA1</span>.2 standard is underspecified by design.
For example:</p>
<ul>
<li>The standard imposes no restriction on the operations returned by the <code>%transfer</code> and <code>%approve</code> entrypoints — since e.g. a contract may need to invoke calls to another contract to access the contents of its ledger, if these are stored remotely.</li>
<li>A contract may contain entrypoints other than those mentioned in the standard (e.g. to mint and burn tokens).</li>
</ul>
<p>Just because such details are not fully specified in <span class="caps">FA1</span>.2 does not mean we can ignore them in our verification effort!
We saw <a href="#every_entrypoint">one instance of this above</a>: an obviously necessary requirement that every entrypoint must preserve storage validity — even if the entrypoint is not mentioned in the <span class="caps">FA1</span>.2 standard.
Thus, the <span class="caps">FA1</span>.2 standard requires (explicitly or implicitly) some invariants which may apply to <em>all</em> of an implementation, even if — especially if — that implementation has extra bells and whistles.<sup id="fnref:to-be-clear"><a class="footnote-ref" href="#fn:to-be-clear">11</a></sup></p>
<p>So for each particular implementation, we formulate an additional specification file describing in more detail relevant behaviour of the implementation’s entrypoints:</p>
<ul>
<li><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_camlcase_spec.v">camlCase per-implementation file</a></li>
<li><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_edukera_spec.v">Edukera per-implementation file</a></li>
<li><a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_dexter_spec.v">Dexter 2 per-implementation file</a></li>
</ul>
<p>For example: all three specification files above include a further requirement <code>ret_ops = nil</code> that the <code>%transfer</code> and <code>%approve</code> entrypoints return no operations.<sup id="fnref:no-operations"><a class="footnote-ref" href="#fn:no-operations">12</a></sup></p>
<p>For example: in the camlCase and Edukera specification files, we require the <code>%getBalance</code> entrypoint to fail if the owner does not have a balance in the ledger; the Dexter 2 specification file must succeed even in this case and return zero.<sup id="fnref:isowner"><a class="footnote-ref" href="#fn:isowner">13</a></sup></p>
<p>Each such contract-tailored <span class="caps">FA1</span>.2 specification is designed in such a way that it: </p>
<ol type="a">
<li> implies the general <span class="caps">FA1</span>.2 specification, and
<li> fully describes the behavior of the given contract.
</ol>
<p>The latter point implies that for each entrypoint, the returned storage and the list of operations are uniquely identified by the initial environment and storage, and the entrypoint parameter.<sup id="fnref:environment"><a class="footnote-ref" href="#fn:environment">14</a></sup></p>
<p>As <a href="#balances-unchanged">mentioned earlier</a>, we can carry out some important verification once and for all, without recourse to a contract-tailored specification file.
For instance, we can prove that in any contract satisfying the general <span class="caps">FA1</span>.2 specification, the total sum of all tokens remains unchanged by any of the five entrypoints.
For the <strong>view</strong> entrypoints this is of course trivial; establishing this for the <code>%approve</code> and (especially) <code>%transfer</code> entrypoints requires some amount of effort that would otherwise have to be carried out for each implementation separately.
The axioms we imposed earlier as part of the abstract <span class="caps">FA1</span>.2 interface are key here.</p>
<p>In contrast, the Edukera Why3 verification features similar invariants, but as one-off proof obligations; <em>e.g.</em>, the following formal property is part of the <code>%transfer</code> entrypoint specification:</p>
<div class="highlight"><pre><span></span><code><span class="k">forall</span> <span class="n">tokenholder</span> <span class="k">in</span> <span class="n">ledger</span><span class="o">,</span>
<span class="n">tokenholder.holder</span> <span class="o"><></span> <span class="o">%</span><span class="n">from</span> <span class="o">-></span>
<span class="n">tokenholder.holder</span> <span class="o"><></span> <span class="o">%</span><span class="n">to</span> <span class="o">-></span>
<span class="n">before.ledger</span><span class="o">[</span><span class="n">tokenholder.holder</span><span class="o">]</span> <span class="o">=</span> <span class="n">some</span><span class="o">(</span><span class="n">tokenholder</span><span class="o">)</span>
</code></pre></div>
<p>This states that if an account is neither the sender <code>%from</code> nor the recipient <code>%to</code>, then its ledger entry is untchanged by the <code>%transfer</code> entrypoint.</p>
<p><a name="problems"></a></p>
<h2 id="problems-detected-problems-solved"><a class="toclink" href="#problems-detected-problems-solved">Problems detected, problems solved</a></h2>
<p>Our verification revealed two discrepancies between how the three implementations above handled corner cases, which had not been explicitly addressed in the the <span class="caps">FA1</span>.2 standard — in other words, the implementations satisfied the <span class="caps">FA1</span>.2 standard but this standard was underspecified.</p>
<h3 id="discrepancy-1-resolved-making-a-transfer-to-yourself"><a class="toclink" href="#discrepancy-1-resolved-making-a-transfer-to-yourself">Discrepancy 1 resolved (making a transfer to yourself)</a></h3>
<p>When the <strong>from</strong> and <strong>to</strong> accounts in the <code>%transfer</code> entrypoint coincide, the operation can be treated either </p>
<ol>
<li>as a <span class="caps">NOOP</span>, or </li>
<li>as a regular transfer.</li>
</ol>
<p>We noted that the camlCase contract implementation treats the operation as a <span class="caps">NOOP</span>; whereas the Edukera and Dexter 2 contract implementations treat the operation as a regular transfer.</p>
<p>There is a real logical difference between these two options.
In the case of a <span class="caps">NOOP</span>, nothing changes at all.
In the case of a regular transfer, the account balances don’t change <em>but</em> the spending allowance <em>does</em>, because it gets decremented by the amount transferred (from the contract to itself).</p>
<p>Following discussions:</p>
<ul>
<li>The <span class="caps">FA1</span>.2 standard was updated to eliminate this ambiguity by requiring that this corner case be treated as a regular transfer (<a href="https://gitlab.com/tzip/tzip/-/issues/43">GitLab issue</a>).</li>
<li>The camlCase implementation of the <code>%transfer</code> entrypoint was updated (<a href="https://gitlab.com/camlcase-dev/fa1.2/-/issues/2">GitLab issue</a>).</li>
</ul>
<h3 id="discrepancy-2-resolved-the-view-entrypoints"><a class="toclink" href="#discrepancy-2-resolved-the-view-entrypoints">Discrepancy 2 resolved (the view entrypoints)</a></h3>
<p>Recall <a href="#view">the <strong>view</strong> entrypoints</a> <code>%getBalance</code>, <code>%getAllowance</code>, and <code>%getTotalSupply</code> mentioned above.
The callback transactions are now required to forward to the callback <em>all of the tez</em> passed to the entrypoint (see <a href="https://gitlab.com/tzip/tzip/-/issues/44">the GitLab issue</a>).
The Edukera and the camlCase implementations were duly updated:</p>
<ul>
<li>The Edukera view entrypoints were updated (<a href="https://github.com/edukera/archetype-lang/commit/26ce017389b0dd8bb688359414cf891c4e287893">GitLab commit</a>). </li>
<li>The camlCase view entrypoints also needed slight adjustment (<a href="https://gitlab.com/camlcase-dev/fa1.2/-/issues/1">GitLab issue</a>).</li>
<li>The Dexter 2 implementation needed no change. The view entrypoints are hardcoded to send zero tez to the callback, and this is compliant since every Dexter 2 <span class="caps">FA1</span>.2 entrypoint is designed to fail if the number of tez passed to it is greater than zero. </li>
</ul>
<h2 id="conclusions"><a class="toclink" href="#conclusions">Conclusions</a></h2>
<p>The formal verification of an <span class="caps">FA1</span>.2 smart contract in Coq has the following components:</p>
<ol>
<li>
<p><em>Checking correctness with respect to the abstract <span class="caps">FA1</span>.2 interface and specification.</em></p>
<p>This includes instantiating concrete Coq types for <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v#L68">the parameter and storage type parameters</a> <code>parameter_ty</code> and <code>storage_ty</code>, providing methods for manipulating them, and proving that <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v#L154">the axioms required by the interface</a> are satisfied. </p>
<p>(Here’s the relevant <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_camlcase.v">file for our first example of the camlCase contract</a>.)</p>
</li>
<li>
<p><em>Formulating a contract-tailored <span class="caps">FA1</span>.2 specification for the implementation’s behaviour — including any additional entrypoints — together with a proof that the general <span class="caps">FA1</span>.2 specification is satisfied.</em></p>
<p>The contract-tailored specification may reference additional data, operators, or behaviour that are not present in the <span class="caps">FA1</span>.2 standard: for instance, some contracts offer an option to <strong>pause</strong> some of the contract’s capabilities; if this flag is set, some (or even all) of the <span class="caps">FA1</span>.2 entrypoints might fail by default.</p>
</li>
<li>
<p><em>Verifying that the contract-tailored <span class="caps">FA1</span>.2 specification does indeed capture the behavior of the contract.</em></p>
<p>Specifically: each of the <span class="caps">FA1</span>.2 entrypoints successfully returns an updated storage and a list of operations if and only if these satisfy the specification.
We don’t need to establish both directions of the if-and-only-if explicitly.
It suffices to prove:</p>
<p><ol type="i">
<li> If an entrypoint successfully returns an updated storage and a list of operations, then these satisfy the specification.
<li> The specification uniquely identifies the return storage and the list of operations. <br>
<li> If there exists a return storage and a list of operations that satisfy the specification, then the entrypoint succeeds (irrespective of the actual storage and list of operations returned).
</ol></p>
<p>Here i is one direction of the equivalence, and i together with ii and iii implies the full equivalence. Proving ii and iii is often simpler than establishing the converse to i directly. </p>
</li>
<li>
<p><em>Verifying that all entry points (including any additional ones) preserve the validity of the storage.</em></p>
<p>For entrypoints outside of <span class="caps">FA1</span>.2 this has to be established explicitly.
For the <span class="caps">FA1</span>.2 entrypoints we already know that the sum of all tokens remains unchanged; hence we only need to show that the <code>%getTotalSupply</code> value remains the same. In case of the Dexter 2, Edukera, and camlCase implementations this was straightforward.</p>
</li>
</ol>
<p>We will continue to improve and simplify the Coq code (and the Mi-Cho-Coq tool on which it depends), and future work includes:</p>
<ol>
<li>An analogous treatment of <a href="https://gitlab.com/tzip/tzip/-/blob/master/proposals/tzip-12/tzip-12.md">the <span class="caps">FA2</span> standard</a>, an instance of which can be plugged into <a href="#dexter2">Dexter 2</a>.</li>
<li><a href="https://gitlab.com/tezos/tezos/-/merge_requests/2297">Adding native <span class="caps">FA1</span>.2 support</a> to <a href="https://tezos.gitlab.io/introduction/howtouse.html#read-the-manual"><code>tezos-client</code></a>. </li>
</ol>
<div class="footnote">
<hr>
<ol>
<li id="fn:swallow">
<p>— and loads of great Coq, of course — <a class="footnote-backref" href="#fnref:swallow" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:it">
<p>E.g. the authors of this blog post may store some value as Euro in a bank account, and some as tez on the Tezos blockchain.
Tokens are what makes a blockchain more than an amusing exercise in distributed computing and databases.
They put the “and” in the sentence: “Blockchain T can let you do X, Y, and Z …. <em>and</em> you get a token for it”. <a class="footnote-backref" href="#fnref:it" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:erc20">
<p>It is similar to the <a href="https://eips.ethereum.org/EIPS/eip-20"><span class="caps">ERC20</span> standard for Ethereum</a>. <a class="footnote-backref" href="#fnref:erc20" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:fa12tzip">
<p>Anyone can propose an improvement to Tezos as a <span class="caps">TZIP</span>.
See the <a href="https://tzip.tezosagora.org"><span class="caps">TZIP</span> explorer</a>, and the <a href="https://gitlab.com/tzip/tzip/-/tree/master/proposals"><span class="caps">TZIP</span> GitLab repo</a>.
The <a href="https://gitlab.com/tzip/tzip/-/blob/master/proposals/tzip-7/tzip-7.md"><span class="caps">TZIP</span> improvement proposal for <span class="caps">FA1</span>.2 is here</a> and the <a href="https://tzip.tezosagora.org/proposal/tzip-7/">rendered markdown is here</a>.</p>
<p><span class="caps">FA1</span>.2 itself builds on the previous (now deprecated) <a href="https://tzip.tezosagora.org/proposal/tzip-5/"><span class="caps">FA1</span> abstract ledger interface</a>. The <a href="https://gitlab.com/tzip/tzip/-/blob/master/proposals/tzip-5/tzip-5.md"><span class="caps">TZIP</span> improvement proposal for <span class="caps">FA1</span> is here</a>. <a class="footnote-backref" href="#fnref:fa12tzip" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:approve-transfer">
<p>In a little more detail: an owner <code>O</code> can issue an authorisation that some spender <code>S</code> may withdraw — i.e. transfer to some other account(s) — up to some amount <code>A</code> of tokens out of <code>O</code><span class="quo">‘</span>s account. This can happen in a single transaction of (up to) <code>A</code>, or in multiple smaller transactions of total no greater than <code>A</code>. <a class="footnote-backref" href="#fnref:approve-transfer" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:second">
<p>Royalty faces similar naming issues. E.g. King Richard the Lionheart only became King Richard I when King Richard <span class="caps">II</span> came along. We recently <a href="https://research-development.nomadic-labs.com/dexter-decentralized-exchange-for-tezos-formal-verification-work-by-nomadic-labs.html">verified the functional correctness of Dexter 1</a>, as part of a larger effort to construct a fully-verified token system. The verification of <span class="caps">FA1</span>.2 discussed in this blog post complements that work. <a class="footnote-backref" href="#fnref:second" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:garbage-in">
<p>Let’s emphasise this point: if we take a bad requirement — or one that has been misunderstood or applied out of context — and formalise it and write code that sanctifies this formalisation with a correct implementation — then all we have done is gone from bad to worse. This sounds obvious but surprisingly often people forget that just because a system is operating normally and as per spec, does not <em>in itself</em> mean that the outcomes are sensible, desirable, or good. This is of course a systems phenomenon that is not restricted to code. <a class="footnote-backref" href="#fnref:garbage-in" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn:annot">
<p><code>getBalance</code> also appears <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_interface.v#L30">inside the <code>annot</code> module</a>. This is a different namespace and can be ignored; similarly for <code>getAllowance</code> and <code>getTotalSupply</code>. <a class="footnote-backref" href="#fnref:annot" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
<li id="fn:others">
<p><a href="#additional_entrypoints">As discussed, additional non-<span class="caps">FA1</span>.2 entrypoints</a> might exist to change the total sum of balances, and this is fine. <a class="footnote-backref" href="#fnref:others" title="Jump back to footnote 9 in the text">↩</a></p>
</li>
<li id="fn:does-not">
<p>To be more precise it says: </p>
<blockquote>
<p><code>%getBalance</code> and <code>%getTotalSupply</code> entrypoints have the same semantics as they do in <span class="caps">FA1</span></p>
</blockquote>
<p>The <span class="caps">FA1</span> standard <a href="https://tzip.tezosagora.org/proposal/tzip-5/#gettotalsupply">tzip-5</a> then says: </p>
<blockquote>
<p><strong>getTotalSupply</strong> This view returns the sum of all participants’ balances.</p>
</blockquote>
<p><a class="footnote-backref" href="#fnref:does-not" title="Jump back to footnote 10 in the text">↩</a></p>
</li>
<li id="fn:to-be-clear">
<p>To really spell this out: choosing what invariants to impose and on what entrypoints, is a creative process that requires and expresses our understanding of the code’s intended meaning. For instance, we (presumably) want all entrypoints to preserve validity of storage, but we might want to allow some (non-<span class="caps">FA1</span>.2) entrypoints to mint or burn tokens. Thus, not only does designing these invariants require us to understand what the entrypoints are supposed to do — but at an even deeper level, designing invariants is a way to formally capture, express, and record this understanding for our colleagues, readers, and any continuous integration tools. This is not new: well-chosen invariants, like well-chosen tests, are how good programmers design good software. <a class="footnote-backref" href="#fnref:to-be-clear" title="Jump back to footnote 11 in the text">↩</a></p>
</li>
<li id="fn:no-operations">
<p>camlCase: the <code>ret_ops = nil</code> on lines
<a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_camlcase_spec.v#L63">63</a>
and
<a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_camlcase_spec.v#L86">86</a>;
Edukera: similarly on lines
<a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_edukera_spec.v#L61">61</a>
and
<a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_edukera_spec.v#L82">82</a>;
Dexter 2: similarly on lines
<a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_dexter_spec.v#L62">62</a>
and
<a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_dexter_spec.v#L84">84</a>. <a class="footnote-backref" href="#fnref:no-operations" title="Jump back to footnote 12 in the text">↩</a></p>
</li>
<li id="fn:isowner">
<ul>
<li>For camlCase this is the condition <code>isOwner sto owner</code> on <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_camlcase_spec.v#L108">line 108 of the spec file</a>, where <code>isOwner</code> is defined <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/blob/kristina-fa12-verification-rebase/src/contracts_coq/fa12_interface.v#L127">here, in the <span class="caps">FA1</span>.2 interface file</a> and states that the owner must have a balance in the ledger.</li>
<li>Similarly for the <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_edukera_spec.v#L104">Edukera spec file, line 104</a>.</li>
<li>Dexter 2 has no <code>isOwner</code> clause; see <a href="https://archive.softwareheritage.org/browse/revision/d116674d67eca4b4b15d7c08efd878a8c8aa864b/?origin_url=https://gitlab.com/nomadic-labs/mi-cho-coq&path=src/contracts_coq/fa12_dexter_spec.v#L123">line 123</a>.
The result of <code>getBalance</code> is returned, which is <code>0</code> when the owner does not have an account.</li>
</ul>
<p><a class="footnote-backref" href="#fnref:isowner" title="Jump back to footnote 13 in the text">↩</a></p>
</li>
<li id="fn:environment">
<p><em>Geek note.</em> This is slightly tautological of course: all we are saying is that the smart contrat’s output depends on everything the smart contract’s output depends on. This includes quantities such as:</p>
<ul>
<li>The <code>AMOUNT</code> sent to the contract.</li>
<li>The contract’s <code>BALANCE</code>.</li>
<li>The state of the blockchain (<code>NOW</code> and <code>LEVEL</code>).</li>
<li>The existence or non-existence of called contracts.</li>
<li>Loads of other stuff. </li>
</ul>
<p>… all of which is rolled up conceptually into “the state of the blockchain” or the “environment of the smart contract when called”. But that’s fine; this is a stateful system so that’s what we would expect. <a class="footnote-backref" href="#fnref:environment" title="Jump back to footnote 14 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Five questions to Nomadic Labs PhDs — Guillaume Bau2021-06-11T15:00:00+02:002021-06-11T15:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-06-11:/five-questions-to-nomadic-labs-phds-guillaume-bau.html<p>Short interview with Guillaume Bau, our new PhD student</p><p>At Nomadic Labs we are proud to create next-gen software … but we are even prouder to help create the next generation of software <em>scientists</em>!</p>
<p>So it’s our great pleasure to host and work with several PhD students, who bring unique perspectives on the technology which they help us to develop, and whose interest in blockchain we are happy to nurture and inform. </p>
<p>In this blogpost, we will ask five questions of one of our students, <strong>Guillaume Bau</strong> (and a couple of questions of his supervisor at Nomadic Labs).</p>
<p>Over to you Guillaume … no the mic is over here … that’s right, you’re on air!</p>
<h1 id="questions-for-guillaume"><a class="toclink" href="#questions-for-guillaume">Questions for Guillaume</a></h1>
<h2 id="1-please-present-yourself-and-your-academic-background"><a class="toclink" href="#1-please-present-yourself-and-your-academic-background">1. Please present yourself and your academic background</a></h2>
<p>I’m currently a PhD student at <a href="https://www.lip6.fr/"><span class="caps">LIP6</span></a> and <a href="https://www.nomadic-labs.com/">Nomadic Labs</a>.
I graduated with a computer science BSc in 2007 from the <a href="http://www.sorbonne-universite.fr/">Université Pierre et Marie Curie (now Sorbonne Université)</a>.
I worked for some time in some network analysis and cybersecurity companies, mainly as a C language developer.
In 2016, I entered a startup targeting floating point manipulations errors and approximations.
This was the occasion for me to use my theoretical background: I developed an abstract interpreter for C and Ada, designed to detect floating point arithmetic misbehaviour in avionics software.</p>
<p>Then, while first entering Nomadic in October 2018, I had the opportunity to continue my academic studies: I graduated with a master’s degree and worked on a prototype for an abstract interpreter for Michelson as my Nomadic Labs internship.</p>
<h2 id="2-tell-us-more-about-the-topic-of-your-phd-who-is-your-mentor-and-what-are-your-research-objectives"><a class="toclink" href="#2-tell-us-more-about-the-topic-of-your-phd-who-is-your-mentor-and-what-are-your-research-objectives">2. Tell us more about the topic of your PhD: who is your mentor, and what are your research objectives?</a></h2>
<p>The topic of my PhD is to develop a static analysis tool for the <a href="https://www.michelson.org">Michelson smart contract programming language</a>, helping developers to assert correctness properties in their Michelson code and to detect errors in it.</p>
<p>It will use <strong>Abstract Interpretation</strong>, a static analysis technique which tends to give quick results, even for thousand-line contracts, and which does not require user intervention or programming to guarantee some properties, as are required with proof assistants.<sup id="fnref:intervention"><a class="footnote-ref" href="#fn:intervention">1</a></sup>
Abstract interpretation works by abstracting known, unknown, and partially-known values as overapproximations, e.g. using an interval [1,10] if we know a value might be 1 or might be 10.
This often allows us to deduce useful mathematical or logical properties, for example, “this value is always different from 0”, or “this piece of code cannot be reached”.</p>
<p><strong>Abstract domains</strong>
are responsible for managing Michelson instructions with respect to these value approximations,
and a large part of the work needed to write an abstract interpreter is to develop the abstract domains handling some specific language feature like Michelson <a href="https://tezos.gitlab.io/active/michelson.html#core-data-types-and-notations"><em>sets</em> or <em>maps</em></a>, or some specific property, like storage changes though multiple calls to a specific smart contract entrypoint.
Of course, because we are always manipulating overapproximations, an analysis can raise false alarms<sup id="fnref:false-positive"><a class="footnote-ref" href="#fn:false-positive">2</a></sup> if the approximation is too loose.</p>
<p>Much of my PhD work goes into developing new abstract domains that make useful trade-offs between precision and performance, to help Tezos smart contracts developers guarantee properties of their work.
For instance, I wrote an abstract domain of “smashed” lists, able to represent as a unique abstract value the set of elements contained inside a Michelson list, and an improved domain will consist in keeping abstract items of the list separate.</p>
<p>My research director is Antoine Miné from <span class="caps">LIP6</span>, and my mentors are Vincent Botbol and Mehdi Bouaziz from Nomadic Labs. My work integrates with <a href="https://mopsa.lip6.fr/">Mopsa</a>, Antoine Miné’s modular framework for Abstract Interpretation.
This is currently maintained by Antoine with the help of the <a href="https://mopsa.lip6.fr/">Mopsa team</a>, and allows building analysers for multiple languages (C and Python analysis are provided built-in), while providing useful primitives and a robust framework for abstraction compositions.</p>
<h2 id="3-what-is-the-value-added-and-innovative-side-of-your-phd-thesis-if-applicable-in-the-tezos-blockchain"><a class="toclink" href="#3-what-is-the-value-added-and-innovative-side-of-your-phd-thesis-if-applicable-in-the-tezos-blockchain">3. What is the value added and innovative side of your PhD thesis (if applicable, in the Tezos blockchain)?</a></h2>
<p>Michelson has an interesting combination of language features.
It is high-level in some respects — it’s a strictly-typed language with high-level data structures — yet low-level in others — e.g. it is stack-based.
Stack-based languages are uncommon<sup id="fnref:runtime"><a class="footnote-ref" href="#fn:runtime">3</a></sup> and there is relatively little work on static analysis for them, and furthermore,
Michelson’s high-level data structures create opportunities to develop rather precise customised abstractions.</p>
<p>Michelson is also interesting because it’s a <em>smart contract</em> language, not a regular programming language, so you tend to write different kinds of programs, for which you may need to prove different kinds of correctness properties.
Some of these may be new and specific to smart contract programming languages; like token-preserving invariants or resources analysis (gas, storage size evolution, and so on).</p>
<p>Interesting properties also emerge from the underlying blockchain platform, e.g. properties of inter-contract and transactions behaviour, and these can depend on the underlying blockchain implementation.<sup id="fnref:ordering"><a class="footnote-ref" href="#fn:ordering">4</a></sup>
That is: if you have a <em>specific</em> blockchain implementation C running smart contracts X, Y, and Z, then an effective analysis of the (full panoply of possible) interactions between smart contracts X, Y, and Z in the context of C, may be key to proving safety, even if you personally only care about the correctness of contract X!
As you might imagine this can get quite complicated, which is why effective automated tools can be so valuable.</p>
<p>Thus from the point of view of a hands-on real-life Tezos smart contract developer, an automatic static analysis tool would add real value.
It would help find errors earlier in you development cycle, and could prevent blocked contracts or malicious exploitation.
Abstract interpretation is scalable to real-life use-cases and does not require much human work to obtain its guarantees of properties.
I would argue that for the working programmer, this is a desirable combination of features!</p>
<h2 id="4-what-is-the-benefit-of-preparing-a-phd-and-spending-most-of-your-time-in-a-private-company-vs-a-university-lab"><a class="toclink" href="#4-what-is-the-benefit-of-preparing-a-phd-and-spending-most-of-your-time-in-a-private-company-vs-a-university-lab">4. What is the benefit of preparing a PhD and spending most of your time in a private company vs. a University lab?</a></h2>
<p>University labs are fecund environments and ideas can be exchanged and relations between different topics explored — but it’s not always the case that the actual developers of the system you’re studying are working in the very same office.
It would be like working on a C language analyser and being able to pop over to Brian Kernighan in the neighboring office to ask “Why did you do it like that?”</p>
<p>Studying at Nomadic, I’m surrounded by Tezos developers, some of whom are working on techniques for certifying smart contracts with Coq; some of whom are working on the Michelson interpreter; and we are all reviewing code from other Tezos contributors.
So I’m right at the heart of Michelson’s development, with the ability to pre-empt the language’s evolution, and even participate in it.</p>
<h2 id="5-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs"><a class="toclink" href="#5-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs">5. Why did you choose to become a part of the blockchain ecosystem and Nomadic labs?</a></h2>
<p>Blockchains are an interesting topic, being at the intersection of several important disciplines: distributed systems, networking, cryptography, language design and analysis and more.
So there is much opportunity for new and useful academic research.</p>
<p>And, the Tezos blockchain in particular considered safety in its design right from the start: the main node is written in a memory-safe language, the cryptographic routines are formally verified, and Michelson was built without implicit failures.
Nomadic engineers share a desire to apply formal verification to guarantee the inner working of the node and network behaviour.
This culture from Tezos and particularly from Nomadic Labs implied working with the OCaml language, and working on Abstract Interpretation, both of which I’m very interested in.</p>
<p>I have a background in and am familiar with basic abstract interpretation techniques, to e.g. find numeric invariants, but I have a lot more to learn to be able to build systems capable of proving properties involving e.g. authentication and authorisation.
While working with the Mopsa framework, I’m learning how real-life and industrially robust abstract interpreters are developed, with advanced feature sets, like modular analysis and abstraction compositions.</p>
<p>I would be very pleased to be able to make a small contribution to making smart contracts more secure — initially on the Tezos chain, but I hope the research may be transferable to others — and I look forward to studying and learning at Nomadic Labs and creating a practical project which will help real Tezos users to create value, by making things that people want and can (safely) use!</p>
<h1 id="questions-for-guillaumes-nomadic-labs-mentor-vincent-botbol"><a class="toclink" href="#questions-for-guillaumes-nomadic-labs-mentor-vincent-botbol">Questions for Guillaume’s Nomadic Labs mentor Vincent Botbol</a></h1>
<h2 id="what-is-your-input-to-guillaumes-thesis"><a class="toclink" href="#what-is-your-input-to-guillaumes-thesis">What is your input to Guillaume’s thesis?</a></h2>
<p>Ph.D. students are expected to absorb vast amounts of information.
My role is to give Guillaume guidance, advice, and a framework to understand the material — as well as to suggest pertinent literature and references where relevant. </p>
<p>Blockchain is a particularly challenging and exciting space in this regard, because it is inherently interdisciplinary and because, being so new, there is less of a body of knowledge to guide us. </p>
<p>My background includes expertise in formal verification (in particular using abstract interpretation) and in the implementation of static analysers. I also know the Tezos codebase and its ecosystem intimately, having participated in its evolution from even before the mainnet launch back in September 2018.</p>
<p>My technical input is then twofold: </p>
<ul>
<li>I will help Guillaume by proposing techniques to increase performance and precision<sup id="fnref:precision"><a class="footnote-ref" href="#fn:precision">5</a></sup> in his static analyser design, and also </li>
<li>I will guide him towards useful real-life use-cases, to make his work maximally useful for the smart contract developer community.</li>
</ul>
<h2 id="why-is-this-thesis-important"><a class="toclink" href="#why-is-this-thesis-important">Why is this thesis important?</a></h2>
<p>Many of the most expensive errors on blockchains are related to buggy smart contracts.
Formal verification can help critical software development but it remains had to apply: it’s great when it works, but tooling tends to be either incomplete or hard to apply (or both).</p>
<p>Guillaume hopes to create a fully automated analyser.
The user just needs to provide a specification (e.g. “only my tz1 address may remove tokens from this contract”) and the analyser will automatically check whether the specification is met.</p>
<p>By relying on the mathematically sound<sup id="fnref:sound"><a class="footnote-ref" href="#fn:sound">6</a></sup> framework that is abstract interpretation we obtain a strong proven guarantee that if the analysis states that an invariant is true, then it is.<sup id="fnref:true"><a class="footnote-ref" href="#fn:true">7</a></sup></p>
<p>Guillaume’s tool will also be able to detect coding style anti-patterns, dead code and security issues.</p>
<p>The abstract interpretation method Guillaume is using has two strengths. First, it’s great with the numerical (i.e. arithmetic) properties which are common in smart contract code (asset management, <span class="caps">ERC20</span>-like contracts, decentralised exchanges, etc.).
Second, it’s fast: thousands of lines of code can be analysed in a few seconds.</p>
<p>I look forward to integrating this analysis into the Tezos smart contract development process and so helping users and developers in the Tezos ecosystem to catch more bugs, more quickly, in their smart contracts.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:intervention">
<p>To be more precise (and in fairness to proof assistants): </p>
<ul>
<li>there is a highly labour-saving subset of properties for which abstract interpretation tends to not require user intervention or programming (e.g. detecting integer overflows), and </li>
<li>there is another still-useful subset for which abstract interpretation is applicable, but which may require some user intervention e.g. to refine invariants (“user X has permission to transfer the contract tokens” might require us to refine our invariant with a precondition like “user X has address tz1”), and </li>
<li>there is a third subset for which abstract interpretation can not provide useful invariants in general: e.g. because the properties involved are just too complex for a general abstract domain to check. In this case, you <em>could</em> try to build a customised abstract domain to prove a specific instance of this property — but this may just be a case of picking the right tool for the job and it may be quicker and easier to prove your invariant using other means, and then input it into the analyser as an assumption.</li>
</ul>
<p>So abstract interpretation is not a magic wand, but it can feel like that sometimes. <a class="footnote-backref" href="#fnref:intervention" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:false-positive">
<p>This is like a <em>false positive</em> in a medical test: not necessarily bad in and of itself, but it wastes time. <a class="footnote-backref" href="#fnref:false-positive" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:runtime">
<p>May I expand on this in a footnote please?
<em>[Sure, please do -ed]</em> </p>
<p>Stack-based general programming languages are uncommon, but low-level assembly and bytecode languages are frequently stack-based: e.g. Intel and <span class="caps">ARM</span> assembly, Java <span class="caps">VM</span> bytecode, Microsoft <span class="caps">CLR</span> bytecode, and OCaml bytecode.</p>
<p>So Michelson is an interesting engineering hybrid that combines features typical of a low-level compile-target language, with features of high-level programming languages. It is possible to program directly in Michelson, though I wouldn’t want to myself.
A key point is that verifying low-level languages requires to reconstruct the programmer’s intent to some degree, which can be hard in some cases and may lead to loss of precision.
But, Michelson’s high-level features ameliorate this, and furthermore, being strictly-typed means that specific classes of problems can never arise, which facilitates the writing of both an interpreter and an abstract interpreter. <a class="footnote-backref" href="#fnref:runtime" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:ordering">
<p>For instance, transaction evaluation ordering differs between Ethereum and Tezos, and Tezos has notions of internal vs. external operations. A good abstract interpreter for Tezos, if it is to be maximally useful, may need to know about and account for these specificities. <a class="footnote-backref" href="#fnref:ordering" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:precision">
<p><em>More performance</em> = <em>Increased speed</em>. <em>More precision</em> = <em>the analysis raises fewer false positive alarms</em>. <a class="footnote-backref" href="#fnref:precision" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:sound">
<p><em>Sound</em> means something rather precise in mathematics, namely “correctness with respect to a logical model that is proven to mathematical precision” …. <a class="footnote-backref" href="#fnref:sound" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:true">
<p>… and <em>correct</em> means “the stated invariants are maintained and there can be no exceptions, counterexamples, or corner cases”. Knowing how to translate intuitive notions of correctness into verifiable invariants is a key skill, but that’s a whole other discussion. <a class="footnote-backref" href="#fnref:true" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
</ol>
</div>Progress report on the verification of Liquidity Baking smart contracts2021-06-04T12:00:00+02:002021-06-04T12:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-06-04:/progress-report-on-the-verification-of-liquidity-baking-smart-contracts.html<h2 id="context"><a class="toclink" href="#context">Context</a></h2>
<p>Liquidity baking is one of the main features of <a href="https://research-development.nomadic-labs.com/announcing-granada.html">the Granada proposal</a>. The feature itself has been
well explained by others (<a href="https://forum.tezosagora.org/t/liquidity-mining-on-tezos/2529">initial proposal on Agora</a>, <a href="https://gitlab.com/tzip/tzip/-/blob/master/drafts/current/draft-liquidity_baking.md"><span class="caps">TZIP</span></a>, <a href="https://midl-dev.medium.com/liquidity-baking-in-granadanet-how-it-works-e60c82bee6f1">Presentation by Midl-dev</a>, <a href="https://xtz.news/category/defi-news/liquidity-baking/">Presentation by <span class="caps">XTZ</span>.news</a>) so we won’t present it here in details.</p>
<p>An original aspect of this feature is that it depends on a smart contract that will likely hold a very high balance. For this reason, the security of this smart contract is extremely important and we have devoted a lot of efforts into checking that the contract is safe.</p>
<p>We have used formal verification to prove that the compiled Michelson code for the <span class="caps">CPMM</span> contract is valid with respect to functional specification. As was shown in the past, this approach increases our confidence in the correctness of the implementation, but says little about the properties of its specification. In particular, <a href="https://research-development.nomadic-labs.com/a-technical-description-of-the-dexter-flaw.html">the specification of Dexter was flawed</a>. To avoid similar pitfall, we also have proven higher-level, security properties for the <span class="caps">CPMM</span> contract, and we have subjected the overall Liquidity Baking feature to thorough testing.</p>
<h2 id="overview-smart-contracts-involved-in-liquidity-baking"><a class="toclink" href="#overview-smart-contracts-involved-in-liquidity-baking">Overview: smart contracts involved in Liquidity Baking</a></h2>
<p>Three smart contracts are involved in the Liquidity Baking feature: a Constant Product Market Maker (<span class="caps">CPMM</span> for short), one <span class="caps">FA1</span>.2 contract managing a pool of <a href="https://tzbtc.io/">tzBTC</a>, and another <span class="caps">FA1</span>.2 contract managing liquidity tokens.</p>
<p>The main contract is the <span class="caps">CPMM</span>; a kind of contract deeply inspired by <a href="https://uniswap.org/">Ethereum’s Uniswap decentralized exchange</a>. Two kinds of users interact with this contract.</p>
<p>The first kind of users are called traders; they can use the contract to trade tez for a particular <span class="caps">FA1</span>.2 token called <a href="https://tzbtc.io">tzBTC</a> (using the <code>xtzToToken</code> entrypoint) and vice versa they can trade tzBTC tokens for tez (using the <code>tokenToXtz</code> entrypoint). Traders can also use the <span class="caps">CPMM</span> to trade tzBTC tokens for other <span class="caps">FA1</span>.2 tokens (using the <code>tokenToToken</code> entrypoint).</p>
<p>In order to perform the trade without waiting for another trader wanting to do the symmetric trade, the <span class="caps">CPMM</span> contract holds both tez and tzBTC tokens that are provided by the second kind of users, called liquidity providers. By maintaining in its storage the number of tokens available, the <span class="caps">CPMM</span> is able to compute an exchange rate for the current trade (independently from the other trades).</p>
<p>Liquidity providers deposit tez and tzBTC tokens simultaneously in a so-called liquidity pool (using the <code>addLiquidity</code> entrypoint). The share of the liquidity pool that each liquidity provider owns is materialized using a <span class="caps">FA1</span>.2 token contract (called the liquidity token contract) administered by the <span class="caps">CPMM</span>. Each time <code>addLiquidity</code> is called new liquidity token are minted and credited to the liquidity provider that called it. Liquidity providers can also withdraw an amount of tez and tzBTC token by burning their liquidity tokens (using the <code>removeLiquidity</code> entrypoint).</p>
<p>Finally, anyone can make a donation in tez to the <span class="caps">CPMM</span> contract (using the <code>default</code> entrypoint). This donation goes to the tez part of the liquidity pool but no liquidity token is minted for it so users have no reason to call this entrypoint; the sent tez are effectively shared between the liquidity providers. The <code>default</code> entrypoint is used by the Tezos protocol to send the liquidity baking subsidy (2.5 freshly minted tez) to the contract in each block.</p>
<p>This subsidy is meant to incentivize liquidity providers. To further incentivize them, for each trade, a 0.1% fee of the traded amount is also sent to the liquidity pool.</p>
<h2 id="verified-properties"><a class="toclink" href="#verified-properties">Verified properties</a></h2>
<p>For each property, we will explain the potential risk if it did not hold, we give an informal description of the checked property, we show the verification technique used to check that this property holds, and we provide pointers to online resources for more details.</p>
<h3 id="notations"><a class="toclink" href="#notations">Notations</a></h3>
<ul>
<li><span class="math">\(L\)</span> : number of liquidity tokens accounted in the <span class="caps">CPMM</span> storage.</li>
<li><span class="math">\(X\)</span> : number of xtz tokens accounted in the <span class="caps">CPMM</span> storage.</li>
<li><span class="math">\(T\)</span> : number of tzBTC tokens accounted in the <span class="caps">CPMM</span> storage.</li>
<li><span class="math">\(\gamma\)</span> is equal to 1 minus the fee (a constant equal to <span class="math">\(0.999\)</span> in the <span class="caps">CPMM</span>).</li>
</ul>
<p>We write <span class="math">\(L'\)</span>, <span class="math">\(X'\)</span>, and <span class="math">\(T'\)</span> for these values after the execution of the entrypoint under focus.</p>
<p><span class="math">\(X\)</span> and <span class="math">\(T\)</span> are called the <em>supplies</em> of the <span class="caps">CPMM</span> and their product <span class="math">\(X \cdot T\)</span> is called <em>the product of supplies</em>.</p>
<h3 id="verification-tools"><a class="toclink" href="#verification-tools">Verification tools</a></h3>
<p>Our main verification tool is the <a href="https://gitlab.com/nomadic-labs/mi-cho-coq">Mi-Cho-Coq</a> library that allows to verify Michelson programs with the <a href="http://coq.inria.fr">Coq proof assistant</a>. Coq is a very expressive system in which essentially any mathematical theorem can be stated and proved. However, Mi-Cho-Coq has some limitations, the most important one in the context of this work is that it cannot currently be used to prove properties about contract interactions.</p>
<p>In that situation, to get more confidence in the validity of properties where Mi-Cho-Coq cannot be used to get a mechanically verified mathematical proof, we use a testing framework based on property-based random testing named <a href="https://github.com/c-cube/qcheck">QCheck</a>. We have extended the test framework which has been developed to test Tezos protocols in two directions: (1) execute arbitrary sequences of Liquidity Baking contract calls on top of arbitrary contexts, and (2) simulate the execution of these contracts.</p>
<p>Both calling the real contracts and simulating them have benefits. Executing the real contracts improves our confidence in the tests results, but has proven to be time-expensive. Simulating the execution addresses this limitation and allows to execute many more test cases, but the results obtained with this approach are only valid if the simulation is trustworthy.</p>
<p>To get the best of both worlds, we have validated our simulation against the contracts implementation, by executing the same sequences of contracts calls using both approaches. During this process, we check after each baked blocks if the simulation state is an abstraction of the blockchain context. In the process, we have both ironed out bugs from our simulation and improved our confidence in the contract implementation.</p>
<h3 id="correctness-of-compilation"><a class="toclink" href="#correctness-of-compilation">Correctness of compilation</a></h3>
<p>The <span class="caps">CPMM</span> and the <span class="caps">FA1</span>.2 liquidity token contract have been developed using <a href="https://ligolang.org/">the <span class="caps">LIGO</span> language</a> and compiled to Michelson scripts. Compilers are complex programs and bugs inside them are commonly found.</p>
<p><strong>Property</strong></p>
<p>The semantics of the Michelson scripts matches the semantics of the source <span class="caps">LIGO</span> scripts.</p>
<p><strong>Risk</strong></p>
<p>Most compiler bugs are benign but in the worst case a bug in the <span class="caps">LIGO</span> compiler could introduce a security risk if the semantics of the <span class="caps">LIGO</span> scripts was not preserved through compilation.</p>
<p><strong>Verification technique</strong></p>
<p>Using the Mi-Cho-Coq framework and the Coq proof assistant, we formally prove that the Michelson scripts are semantically equivalent to a high-level specification arguably equivalent to the source <span class="caps">LIGO</span> scripts. During this verification work, no bug in the <span class="caps">LIGO</span> compiler was found.</p>
<p><em>Resources:</em></p>
<ul>
<li><a href="http://yann.regis-gianas.org/static/cpmm/cpmm_report.html#compilation-correctness">Coq statements</a></li>
<li><a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/tree/yrg@cpmm2-economic-properties/src/contracts_coq">Coq proof repository</a></li>
</ul>
<h3 id="safety-of-execution"><a class="toclink" href="#safety-of-execution">Safety of execution</a></h3>
<p>All the entrypoints of the <span class="caps">CPMM</span>, except <code>default</code>, involve divisions. We make sure that the script never divides by zero. As a matter of fact, the contracts divide
by the number of tokens of the pools (the pool depending on the entrypoint in question).</p>
<p><strong>Property</strong></p>
<p><span class="math">\(L\)</span>, <span class="math">\(X\)</span>, and <span class="math">\(T\)</span> should always be strictly positive. This is an obvious consequence of the following property:</p>
<div class="math">$$
0 < L^2 \le X \cdot T
$$</div>
<p>In plain english: the square of the total number of liquidity tokens is always positive and the product of xtz tokens and <span class="caps">FA1</span>.2 tokens is always greater or equal to this value.</p>
<p><strong>Risk</strong></p>
<p>If <span class="math">\(L\)</span>, <span class="math">\(X\)</span>, or <span class="math">\(T\)</span> ever reaches zero, some entrypoints of the <span class="caps">CPMM</span> cannot be called anymore so the functionality they provide is lost. In particular, some funds would be frozen and the subsidy would also be minted for nothing.</p>
<p><strong>Verification technique</strong></p>
<p>Using the Mi-Cho-Coq framework and the Coq proof assistant, we formally prove that all the entrypoints preserve this relationship between <span class="math">\(L\)</span>, <span class="math">\(X\)</span>, and <span class="math">\(T\)</span> and that this property is sufficient to show that the token pool never reaches 0 (hence, the contract never divides by zero and never gets stuck).</p>
<p>The case for <code>removeLiquidity</code> is interesting because it can lead to <span class="math">\(L = 0\)</span> in the edge case of all liquidity providers choosing to remove all their liquidities. To guarantee that this case never happens, a tiny initial liquidity deposit is done with the null address so that it can never be removed in the future. This external assumption has been taken into account in the formal proof.</p>
<p><em>Resources:</em></p>
<ul>
<li><a href="http://yann.regis-gianas.org/static/cpmm/cpmm_report.html#safety">Coq statements</a></li>
<li><a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/tree/yrg@cpmm2-economic-properties/src/contracts_coq">Coq proof repository</a></li>
</ul>
<h3 id="evolution-of-the-product-of-supplies"><a class="toclink" href="#evolution-of-the-product-of-supplies">Evolution of the product of supplies</a></h3>
<p>Despite its name, the <span class="caps">CPMM</span> does not preserve a constant product of supplies. Indeed, the <code>default</code> entrypoint is increasing <span class="math">\(X\)</span> with no change to <span class="math">\(T\)</span>. The fees and the rounding in division also prevent the product of supplies to be constant.</p>
<p>As a matter of fact, the product of supplies generally increases. This is true for allthe entrypoints, except, of course, for <code>removeLiquidity</code> that decreases both <span class="math">\(X\)</span> and <span class="math">\(T\)</span>.</p>
<p>In order to take the <code>removeLiquidity</code> case into account, we need to track not only <span class="math">\(X\)</span> and <span class="math">\(T\)</span> but also <span class="math">\(L\)</span>. The correct invariant is a property named “Ratio between Product and Squared Liquidity Increases” and it holds for all entrypoints:
</p>
<div class="math">$$
{ { X \cdot T } \over L^2 } \le { { X' \cdot T' } \over L'^2 } \qquad (RPSLI)
$$</div>
<p>This property is important because it guarantees that there is an incentive for providing liquidity: whatever the users of the contract do, liquidity providers are guaranteed that the amount of tez and tzBTC tokens that they will be able to withdraw in the future always have a greater product than what they originally deposited.</p>
<p><strong>Property</strong></p>
<ul>
<li>
<p><code>addLiquidity</code> of <span class="math">\(A\)</span> tez increases the product of supplies as follows:
<div class="math">$$
X' \cdot T' = (X + A) \cdot (T + {\left\lceil {{T \cdot A} \over X} \right\rceil}) \ge X \cdot T
$$</div>
and <span class="math">\(L' = L + { { L \cdot A } \over X }\)</span>.</p>
</li>
<li>
<p><code>removeLiquidity</code> of <span class="math">\(A\)</span> liquidity tokens decreases the product of supplies as follows:
<div class="math">$$
X' \cdot T' = (X - { { X \cdot A } \over L}) \cdot (T - { { T \cdot A } \over L}) \le X \cdot T
$$</div>
and <span class="math">\(L' = L - A\)</span>.</p>
</li>
<li>
<p><code>xtzToToken</code> of <span class="math">\(A\)</span> tez increases the product of supplies as follows:
<div class="math">$$
X' \cdot T' = (X + A \cdot \gamma) \cdot (T - {{ \gamma \cdot A \cdot T } \over {X + \gamma \cdot A}}) \ge X \cdot T
$$</div>
and <span class="math">\(L' = L\)</span>.</p>
</li>
<li>
<p><code>tokenToXtz</code> of <span class="math">\(A\)</span> tzBTC tokens increases the product of supplies as follows:
<div class="math">$$
X' \cdot T' = (X - { { A \cdot \gamma \cdot X } \over { T + A \cdot \gamma } }) \cdot (T + A) \ge X \cdot T
$$</div>
and <span class="math">\(L' = L\)</span>.</p>
</li>
<li>
<p><code>tokenToToken</code> of <span class="math">\(A\)</span> tzBTC tokens increases the product of supplies as follows:
<div class="math">$$
X' \cdot T' = (X - { { A \cdot \gamma \cdot X } \over { T + A \cdot \gamma } }) \cdot (T + A) \ge X \cdot T
$$</div>
and <span class="math">\(L' = L\)</span>.</p>
</li>
<li>
<p><code>default</code> of <span class="math">\(A\)</span> tez increases the product of supplies as follows:
<div class="math">$$
X' \cdot T' = (X + A) \cdot T \ge X \cdot T
$$</div>
and <span class="math">\(L' = L\)</span>.</p>
</li>
</ul>
<p><strong>Risk</strong></p>
<p>A strict decrease in the product of supplies means that value is taken out of the liquidity pool. The only case where such a decrease is expected is the <code>removeLiquidity</code> entrypoint. A decrease in any other entrypoint means that an attacker has found a way to steal value from the liquidity pool.</p>
<p><strong>Verification technique</strong></p>
<p>Using the Mi-Cho-Coq framework and the Coq proof assistant, we formally prove that each entrypoint satisfies its corresponding property. We also formally proved that each property implies the property <span class="math">\(RPSLI\)</span>.</p>
<p><em>Resources:</em></p>
<ul>
<li><a href="http://yann.regis-gianas.org/static/cpmm/cpmm_report.html#product">Coq statements</a></li>
<li><a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/tree/yrg@cpmm2-economic-properties/src/contracts_coq">Coq proof repository</a></li>
</ul>
<h3 id="consistency-between-the-contracts-internal-state"><a class="toclink" href="#consistency-between-the-contracts-internal-state">Consistency between the contracts’ internal state</a></h3>
<p>As explained in the overview, there are three contracts involved in Liquidity Baking: they must have a consistent image of the token distribution. The <span class="caps">CPMM</span> storage contains the <span class="caps">CPMM</span>’s view on both parts of its the liquidity pool (<span class="math">\(X\)</span> and <span class="math">\(T\)</span>) and also the total supply <span class="math">\(L\)</span> of the liquidity token contract. The liquidity token contract also stores this value alongside the token balances of all the liquidity providers. The tzBTC contract stores the tzBTC balances of all tzBTC holders, in particular the <span class="caps">CPMM</span>.</p>
<p><strong>Properties</strong></p>
<ul>
<li><span class="math">\(L\)</span> should always be equal to the total supply of the liquidity token contract.</li>
<li><span class="math">\(X\)</span> should always be equal to the tez balance of the <span class="caps">CPMM</span> contract,</li>
<li><span class="math">\(T\)</span> should always be smaller or equal to the tzBTC balance of the <span class="caps">CPMM</span> in the tzBTC contract.</li>
</ul>
<p>Remark that <span class="math">\(T\)</span> could become strictly smaller than the tzBTC balance of the <span class="caps">CPMM</span> contract if a tzBTC holders were to directly call the tzBTC contract to transfer tokens to the <span class="caps">CPMM</span> (instead of calling the <code>tokenToXtz</code> or the <code>addLiquidity</code> entrypoint of the <span class="caps">CPMM</span>); in that case the <span class="caps">CPMM</span> contract can not reject the transfer or even discover that the transfer happened. Users have no reason to donate tzBTC tokens to the <span class="caps">CPMM</span> but when reasoning about the security of Liquidity Baking we need to consider this edge case.</p>
<p><strong>Risk</strong></p>
<p>An inconsistency between the states of the three contracts could lead the <span class="caps">CPMM</span> to incorrectly compute exchange rates and liquidity payback.</p>
<p><strong>Verification technique</strong></p>
<p>The test framework of the Tezos protocol has been extended to originate three contracts: the <span class="caps">CPMM</span>, the <span class="caps">FA1</span>.2 contract to deal with liquidity tokens, and another <span class="caps">FA1</span>.2 contract to act as a replacement for tzBTC (the latter being way more complicated to originate than the former).</p>
<p>We generate sequences of calls to the three contracts involved in Liquidity Baking, and apply the transactions on top an initial context provided by the test framework. After each baked block, we check that the contracts’ storage remain consistent with each other, <em>i.e.</em>, (1) the tez balance stored in the ledger, (2) the storage of the <span class="caps">FA1</span>.2 contract, and (3) the storage of the tzBTC replacement contract.</p>
<p><em>Resources:</em></p>
<ul>
<li>The implementation of our property-based tests can be found in <a href="https://gitlab.com/nomadic-labs/tezos/-/commits/arvid@lthms@liquidity-baking-pbt">a dedicated branch in the Nomadic Labs repository of Tezos</a></li>
</ul>
<h3 id="no-global-gain"><a class="toclink" href="#no-global-gain">No global gain</a></h3>
<p>Assuming an attacker with an arbitrary balance of tez, tzBTC, and liquidity token, we do not expect the attacker to be able to gain value through its interactions with the <span class="caps">CPMM</span> contract alone. Each call to an entrypoint of the <span class="caps">CPMM</span> requires a payment in at least one of the assets but we could imagine, if a vulnerability was present in the <span class="caps">CPMM</span> code, that a sequence of calls resulted in the attacker increasing their holding in all three balances. This is what we call a global gain.</p>
<p><strong>Property</strong></p>
<p>For any sequence of calls from a fixed attacker to the <span class="caps">CPMM</span>, the global effect of the sequence of calls on the balances of the attacker in tez, tzBTC, and liquidity tokens is such that at least one of the balances decreases.</p>
<p>Note that we are assuming that the attacker is the only entity interacting with the contract. In fact, if we take the subsidy into account a global gain is possible by adding liquidities, waiting a few blocks (during which the subsidy is received by the <span class="caps">CPMM</span> and shared between the liquidity providers), and finally removing the liquidities. But this global gain coming from the subsidy is not an attack on the <span class="caps">CPMM</span>, it is the purpose of the Liquidity Baking subsidy!</p>
<p><strong>Risk</strong></p>
<p>Even a very small global gain is very dangerous for the security of the contract because an attacker able to exploit a global-gain vulnerability can likely exploit it many times in a row and the total loss for the contract can be dramatic.</p>
<p><strong>Verification technique</strong></p>
<p>We have looked for global-gain attacks using our property-based testing framework. More precisely, the test consists in generating 100 initial contexts, and for each of these contexts, generating and executing a sequence of 100 valid contracts calls performed by the same implicit account <span class="math">\(c\)</span>. Note that the execution is done with a subsidy equal to <span class="math">\(0\)</span>, because the objective of this test is to discover if an attacker can become richer by exploiting the <span class="caps">CPMM</span>, not by depositing funds against liquidity tokens. After each contract call, the balance of <span class="math">\(c\)</span> is checked to assert that either one of its balances (xtz, tzBTC or lqt) has decreased, or they all remained constant.</p>
<p>QCheck has not being able to find a counter-example to this property.</p>
<p><em>Resources</em>:</p>
<ul>
<li><a href="https://gitlab.com/nomadic-labs/tezos/-/commit/13299a604c9d78908be8b1da8ff50844172a5a69">The commit introducing the test</a></li>
</ul>
<h2 id="trust-base"><a class="toclink" href="#trust-base">Trust base</a></h2>
<p>Our verification and property-based testing efforts have mostly been directed to the liquidity token <span class="caps">FA1</span>.2 and <span class="caps">CPMM</span> scripts because these two scripts are new to
the Tezos blockchain; the Granada protocol, if activated, will originate these scripts during the migration from Florence. The third contract involved, the tzBTC <span class="caps">FA1</span>.2 contract, is however already used onchain, it has been <a href="https://github.com/tf-gh/tzBTC/blob/master/Least%20Authority%20-%20TzBTC%20Updated%20Final%20Security%20Audit%20Report.pdf">audited</a> and it is has been traded on Tezos decentralized exchanges (<a href="https://dexter.exchange/">Dexter</a> and <a href="https://quipuswap.com/swap">Quipuswap</a>) for months.</p>
<p>The security of Liquidity Baking relies on the assumption that the bahavior of the tzBTC contract is consistent with the one of the <span class="caps">FA1</span>.2 contract that we have used as a replacement for it in our property-based testing context.</p>
<p>We also assume that Mi-Cho-Coq correctly defines the semantics of the Michelson language. If this was not the case, we could have missed compilation bugs.</p>
<h2 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h2>
<p>Since the very beginning, we were aware Liquidity Baking is a sensitive feature, even more due to the mechanisms implemented to encourage traders to use it. This is why we have devoted a significant amount of effort to increase our confidence in its implementation. The formal verification effort of the <span class="caps">CPMM</span> contract includes key functional properties. The complete architecture (the three contracts, and the changes integrated to the Granada protocol) has been challenged with unit testing and property-based testing. Finally, the proposal comes with two halting mechanisms that further reduce the risk of a malicious actor taking advantage of the <span class="caps">CPMM</span> contract, and let the bakers be the final arbiters of Liquidity Baking’s fate.</p>
<p>Overall, we feel confident that all components of the Liquidity Baking features are sound and secure.</p>
<p>While the <a href="https://tzstats.com/election/head">Granada election</a> takes its course, we plan to continue our effort on asserting the economic properties of Liquidity Baking. The latter are reminiscent of game theory properties where actors aim to maximize their gains by responding to incentives in the contracts. We believe we now have a strong foundation for tackling this additional challenge.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Announcing Granada2021-05-31T08:00:00+02:002021-05-31T08:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-05-31:/announcing-granada.html<p><em>This is a joint post from Nomadic Labs, Marigold, <span class="caps">TQ</span>, Tarides, and DaiLambda.</em></p>
<p>We were proud to see <a href="https://research-development.nomadic-labs.com/florence-our-next-protocol-upgrade-proposal.html">Florence</a> go live on the chain on 11th May 2021. In keeping with our policy of <a href="https://forum.tezosagora.org/t/regular-scheduling-for-our-tezos-proposals/2331">proposing upgrades on a regularly scheduled basis</a>, we’re happy to announce our latest Tezos protocol proposal, <a href="https://tezos.gitlab.io/protocols/010_granada.html"><strong>Granada</strong></a>. </p>
<p>(As is usual, Granada’s “true name” is its hash, which is <code>PtGRANADsDU8R9daYKAgWnQYAJ64omN1o3KMGVCykShA97vQbvV</code>).</p>
<p>Granada contains several major improvements to the protocol, as well as numerous bug fixes and minor improvements. Below we discuss some of the most interesting and important changes:</p>
<ul>
<li>
<p><strong>Emmy*</strong>: The current Tezos consensus algorithm is Emmy+ and we propose to replace it with Emmy*. As described in <a href="https://research-development.nomadic-labs.com/faster-finality-with-emmy.html">this blog post</a>, if Granada is adopted, Emmy* will generally halve the time between blocks, <strong>from 60 seconds to 30 seconds</strong>, and allow transactions to achieve significantly faster finality than under the current consensus algorithm. (We expect several significant further improvements to our consensus algorithm and reductions in block times in coming proposals.)</p>
</li>
<li>
<p><strong>Liquidity Baking</strong>: The availability of low-slippage exchange of tez into other currencies and vice-versa is key to allow the widespread use of Tezos. <a href="https://gitlab.com/tzip/tzip/-/blob/master/drafts/current/draft-liquidity_baking.md">Liquidity Baking</a> addresses this directly by piggybacking off the liquidity and global availability of Bitcoin, and incentivizing large amounts of decentralized liquidity provision between tez and wrapped bitcoins.</p>
</li>
<li>
<p><strong>Gas improvements</strong>: A number of substantial improvements to performance have been made, which in turn result in dramatic reductions in gas consumption. <strong>We have generally observed a decrease of a factor of three to six in the gas consumed in the execution of already deployed contracts.</strong> For some contracts, the improvement has been almost a factor of eight. This reduction in gas consumption, the latest in a series we began with <em>Delphi</em>, will enable developers to deploy richer, more complicated, and more interesting applications on Tezos at reasonable real-world cost. </p>
</li>
</ul>
<p>Granada contains numerous other bug fixes and small improvements, and <a href="https://tezos.gitlab.io/protocols/010_granada.html">we encourage you to look at the changelog</a> for a full description of the contents of the proposal.</p>
<p>We strongly encourage you to test your own Tezos-based applications to check for compatibility problems with Granada. Granada, and the configuration for its test network <em>Granadanet</em>, are <a href="https://tezos.gitlab.io/releases/version-9.html">included in version 9.2 of the Tezos node</a>. </p>
<p>If Granada is adopted, the next proposal (which likely will have a name starting with the letter “H”) should be proposed and enter the Tezos amendment process this summer.</p>
<p>We hope that “H” will introduce a new consensus algorithm that, if adopted, will bring fast finality to Tezos.</p>
<p>Over the course of the coming months, our team also intends to continue to develop and propose amendments to increase performance, lower gas consumption, reduce block times, and increase Tezos’ throughput (as measured, for example, in transactions per seconds, or smart contract invocations per second).</p>Simulating Tenderbake2021-05-21T11:00:00+02:002021-05-21T11:00:00+02:00Mark Karpovtag:research-development.nomadic-labs.com,2021-05-21:/simulating-tenderbake.html<p>Announcing a simulator for Tenderbake</p><p>If you’re impatient, you are welcome to <a href="guide">read the guide</a> and jump right to <a href="https://gitlab.com/nomadic-labs/tenderbake-simulator">the simulator</a> now.
See also the <a href="#implemented">implemented algorithms</a> below. </p>
<p>If you find that reading about it helps you get excited before using your simulator, then read on …</p>
<h2 id="background"><a class="toclink" href="#background">Background</a></h2>
<p>The consensus algorithm is a crucial part of any blockchain project.
Because of the distributed nature of blockchains, different nodes can
have different ideas of what the current state of the blockchain is
supposed to be. The role of the consensus algorithm is to decide which
of these possible states, called <em>forks</em> or <em>branches</em>, will be
selected globally. In the world of distributed systems, there are two
distinctive families of consensus algorithms: <em>Nakamoto-style</em> and
<em>classical <span class="caps">BFT</span>-style</em>. Most blockchain solutions use <strong>Nakamoto-style</strong>
algorithms that allow the existence of any number of forks of any
length, but make longer forks increasingly unstable, so that they
eventually collapse to a single branch. We say that these algorithms
have <em>probabilistic finality</em>. <strong>Classical Byzantine fault tolerance (classical <span class="caps">BFT</span>)</strong>
algorithms refer to a class of consensus algorithms which offer
<em>deterministic finality</em> over asynchronous networks (you can check
these two classical papers for more details <a href="https://www.microsoft.com/en-us/research/publication/byzantine-generals-problem/?from=http%3A%2F%2Fresearch.microsoft.com%2Fen-us%2Fum%2Fpeople%2Flamport%2Fpubs%2Fbyz.pdf">The Byzantine Generals
Problem</a> and <a href="http://pmg.csail.mit.edu/papers/osdi99.pdf">Practical Byzantine Fault Tolerance</a>).
They stipulate definite conditions that must be fulfilled for a block
to become final.</p>
<p>Nomadic Labs intends to propose <strong>Tenderbake</strong> (<a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">blog post</a>; <a href="tenderbake-paper">paper</a>) — a
classical <span class="caps">BFT</span>-style algorithm — as the next consensus algorithm of Tezos.
Its deterministic finality allows us to make solid claims about the period of time that should pass for a transaction to become final.
In Tenderbake, a block becomes final when there are two blocks on top of it; this is the only condition.
So, if the system produces one block per 15 seconds, a transaction will become final in about 30 seconds.
This is the kind of performance that users expect from a successful blockchain solution.</p>
<p>Historically, Tenderbake started as a variant of <a href="https://arxiv.org/pdf/1807.04938.pdf">Tendermint</a> and was subsequently adapted to fit into the existing Tezos system.
However, the description in <a href="https://arxiv.org/pdf/2001.11965.pdf">the Tenderbake paper</a> and its nascent implementation in the Tezos codebase started to drift apart.
Thus, for a person starting working on Tenderbake, reading the paper is not enough.
It is also not ideal to study the algorithm by reading the Tezos code, since the consensus algorithm is by design implemented in close cooperation with other components of the system. </p>
<p>This is where the <a href="https://gitlab.com/nomadic-labs/tenderbake-simulator"><strong>Tenderbake simulator project</strong></a> steps in.</p>
<h2 id="the-simulation-framework"><a class="toclink" href="#the-simulation-framework">The simulation framework</a></h2>
<p>Nomadic Labs asked <a href="https://tweag.io">Tweag</a> to develop a framework that would be
general enough to model any consensus algorithm (be it <span class="caps">BTF</span>-style,
Nakamoto-style or in styles yet to be invented) in a clear way to facilitate
onboarding of newcomers and for exploration of consensus algorithms in the
future. The results of our work can be found <a href="https://gitlab.com/nomadic-labs/tenderbake-simulator">in this repository</a>.
The language of choice at Nomadic Labs is OCaml and it made sense to use it for this project.
The simulator is distributed under the <span class="caps">MIT</span> license, like the Tezos codebase.</p>
<h3 id="principles-of-operation"><a class="toclink" href="#principles-of-operation">Principles of operation</a></h3>
<p>Here’s a (simplified) description that should give the reader a flavour of the simulation framework.
The framework comes with a <a href="https://gitlab.com/nomadic-labs/tenderbake-simulator/-/blob/master/GUIDE.md">guide</a> that explains in detail how the simulation framework works and how to implement a consensus algorithm in it.</p>
<p>The framework allows us to observe the evolution of a system that comprises
a collection of nodes — independent processes that do not share memory but
can exchange messages by means of asynchronous broadcast that can be
fine-tuned by the user if desired.</p>
<p>Consensus algorithms in the framework are implemented as event handlers —
functions that are called when an event occurs at a particular node. A call
of an event handler is called an <em>iteration</em>. Event handlers have the
following type signature:</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="n">event_handler</span> <span class="o">=</span>
<span class="nn">Algorithm</span><span class="p">.</span><span class="n">params</span> <span class="o">-></span>
<span class="nn">Time</span><span class="p">.</span><span class="n">t</span> <span class="o">-></span>
<span class="nn">Event</span><span class="p">.</span><span class="n">t</span> <span class="o">-></span>
<span class="nn">Signature</span><span class="p">.</span><span class="n">private_key</span> <span class="o">-></span>
<span class="nn">Algorithm</span><span class="p">.</span><span class="n">node_state</span> <span class="o">-></span>
<span class="nn">Effect</span><span class="p">.</span><span class="n">t</span> <span class="kt">list</span> <span class="o">*</span> <span class="nn">Algorithm</span><span class="p">.</span><span class="n">node_state</span>
</code></pre></div>
<p>Let’s go over the arguments of the function:</p>
<ul>
<li><code>Algorithm.params</code> is the parameters of the algorithm such as e.g. round
duration in seconds.</li>
<li><code>Time.t</code> is the current time.</li>
<li><code>Event.t</code> is the event the node needs to react to. Currently there are two
kinds of events: reception of a message and a “wake up” call that the node
can schedule for itself. Message types are defined per consensus algorithm.</li>
<li><code>Signature.private_key</code> is the private key that every node magically
knows. It is used for signing of messages. This is important because the
framework allows us to program and use Byzantine versions of nodes, too.</li>
<li><code>Algorithm.node_state</code> is the node state. The type is defined per
consensus algorithm.</li>
</ul>
<p>The return type of an <code>event_handler</code> is an effect list and an updated node
state. An effect in the list can be one of the following:</p>
<ul>
<li>Broadcast a message to all nodes.</li>
<li>Schedule a wake up call.</li>
<li>Shut down.</li>
</ul>
<h3 id="testing"><a class="toclink" href="#testing">Testing</a></h3>
<p>We have written two kinds of tests: <em>stress tests</em> and <em>scenario tests</em>.</p>
<p><strong>Stress tests</strong> are about letting an algorithm run for a number of
iterations with a large enough network and realistic propagation of
messages, including messages getting lost and messages arriving out of
order. The framework allows us to specify a set of predicates that
must hold at each iteration in such a test. This way we can determine
if an algorithm satisfies <a href="https://www.cs.cornell.edu/fbs/publications/RecSafeLive.pdf">liveness and safety</a>
properties (see also the <a href="https://en.wikipedia.org/wiki/Liveness">Liveness</a> and
<a href="https://en.wikipedia.org/wiki/Safety_property">Safety</a> Wikipedia articles). According to the tests, all
models that we have written satisfy both Liveness and Safety.</p>
<p><strong>Scenario tests</strong> are about adjusting propagation of messages and/or
lifetime of nodes in order to model a situation of interest. We can then
inspect execution logs and check whether the nodes behaved in the expected
way. It is easy to do because simulations return typed logs that we can
pattern match on.</p>
<p><a name="implemented"></a></p>
<h3 id="implemented-algorithms"><a class="toclink" href="#implemented-algorithms">Implemented algorithms</a></h3>
<p>We have implemented four consensus algorithms (listed below roughly in order
of increasing complexity), and applied stress tests and scenario tests as
discussed above:</p>
<ul>
<li>Leader election, see
<a href="https://gitlab.com/nomadic-labs/tenderbake-simulator/-/tree/master/src/leader_election"><code>src/leader_election</code></a>.</li>
<li>Ouroboros (the simple <span class="caps">BFT</span> version), see
<a href="https://gitlab.com/nomadic-labs/tenderbake-simulator/-/tree/master/src/ouroboros"><code>src/ouroboros</code></a>.</li>
<li>Emmy<sup>+</sup>, see
<a href="https://gitlab.com/nomadic-labs/tenderbake-simulator/-/tree/master/src/emmy_plus"><code>src/emmy_plus</code></a>;
this is the current consensus algorithm used by Tezos.</li>
<li>Tenderbake, see
<a href="https://gitlab.com/nomadic-labs/tenderbake-simulator/-/tree/master/src/tenderbake"><code>src/tenderbake</code></a>;
this is the algorithm Nomadic Labs is planning to propose as a future
amendment to the Tezos blockchain.</li>
</ul>
<p>Every algorithm is explained in its <code>README.md</code>. Our focus was not only
explaining how a particular algorithm works in principle, but also how it
translates to code that the simulator framework can run.</p>
<h2 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h2>
<p>The simulator has already proven useful. People who have tried it out report
that it has helped them to understand the algorithm of Tenderbake and
experiment with it. In the future, we can expect new consensus algorithms to
be implemented and explored using this framework. Please, feel free to <a href="https://gitlab.com/nomadic-labs/tenderbake-simulator">give it a try</a> and contribute!</p>Five questions to Nomadic Labs PhDs — Colin González2021-05-12T10:00:00+02:002021-05-12T10:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-05-12:/five-questions-to-nomadic-labs-phds-colin-gonzalez.html<p>Short interview with Colin Gonzalez, our new PhD student</p><p>At Nomadic Labs we are proud to create next-gen software … but we are even prouder to help create the next generation of software <em>scientists</em>!</p>
<p>So it’s our great pleasure to host and work with several PhD students, who bring unique perspectives on the technology which they help us to develop, and whose interest in blockchain we are happy to nurture and inform.</p>
<p>In this blogpost, we will ask five questions of one of our students, Colin Gonzalez (and a couple of questions of his supervisor at Nomadic Labs).</p>
<p>Over to you Colin … no the mic is over here … that’s right, you’re on air!</p>
<h1 id="questions-for-colin"><a class="toclink" href="#questions-for-colin">Questions for Colin</a></h1>
<h2 id="1-please-present-yourself-and-your-academic-background"><a class="toclink" href="#1-please-present-yourself-and-your-academic-background">1. Please present yourself and your academic background</a></h2>
<p>Currently I’m a Ph.D. candidate in Computer Science at the Université de Paris and Nomadic Labs.<sup id="fnref:affiliation"><a class="footnote-ref" href="#fn:affiliation">1</a></sup>
From 2011 to 2014 I did a BSc in Computer Science at the Université Paris Diderot, now Université de Paris.
From 2014 to 2015 I took a year to be a Service Civique volunteer, serving in an <span class="caps">NGO</span> called Afev which focuses on non-academic education for disadvantaged children.
From 2015 to 2017 I did an MSc in Algorithms and Software Engineering with a focus on Programming Languages, at the Université Diderot.
During the second year of the MSc, I got a six-month internship with Yann Regis-Gianas at Université Diderot’s Computer Sciences Lab, on the semantics and implementation of a collaborative spreadsheet system.
From 2018 to 2019 I worked as a critical software engineer for the Université Diderot Physics Department.
There I developed my interests and skills in critical systems, using formal methods to develop embedded software for satellites.
In 2019 I joined Nomadic Labs to start my Ph.D.</p>
<h2 id="2-tell-us-more-about-the-topic-of-your-phd-who-is-your-mentor-research-objectives"><a class="toclink" href="#2-tell-us-more-about-the-topic-of-your-phd-who-is-your-mentor-research-objectives">2. Tell us more about the topic of your PhD, who is your mentor, research objectives</a></h2>
<p>The topic of my Ph.D. is building tools to develop smart contracts.
My main mentor is Yann Regis-Gianas at Nomadic Labs, and I also work with Benjamin Canou from <span class="caps">NL</span> and with researchers from the Université de Paris’ Laboratory called <span class="caps">IRIF</span>.</p>
<p>One challenge is to make these tools comfortable for end-users unfamiliar with existing smart contract programming languages like Michelson or Ligo.
In particular, we see a potential user base in the spreadsheet user community.</p>
<p>The obvious question arises: could we use spreadsheets to design and to build <em>safe</em> and <em>efficient</em> smart contracts?
Safety is critical for smart contracts, because they inherently tend to be used in safety-critical applications, and also because by design, their code cannot be modified once they are launched. We intend to use certified compilation and formal methods to make a trustworthy tool.</p>
<p>With respect to efficiency, we intend to use the results of the <span class="caps">ESOP</span>‘19 publication, <em>Incremental Lambda-Calculus in Cache Transfer Style</em> by Paolo G. Giarrusso, Yann Regis-Gianas and their coauthors (the <a href="https://inc-lc.github.io">incremental lambda-calculus webpage</a> includes the paper and supplementary material).
The techniques of the paper could be used to automatically incrementalize smart contracts, improving their efficiency and shortening execution time.</p>
<h2 id="3-what-is-the-value-added-and-innovative-side-of-your-phd-thesis-if-applicable-in-the-tezos-blockchain"><a class="toclink" href="#3-what-is-the-value-added-and-innovative-side-of-your-phd-thesis-if-applicable-in-the-tezos-blockchain">3. What is the value added and innovative side of your PhD thesis (if applicable, in the Tezos blockchain)?</a></h2>
<p>Currently, smart contracts are written in text-based programming languages.
People who know programming know how to do this, but what about non-programmers?</p>
<p>Hopefully, a spreadsheet-based tool could make the creation of safe smart contracts more accessible to end users.
This could lead a large community of users who are already familiar with spreadsheets being able to launch smart contracts without needing expertise in a `traditional’ programming language.</p>
<h2 id="4-what-is-the-benefit-of-preparing-a-phd-and-spending-most-of-your-time-in-a-private-company-vs-a-university-lab"><a class="toclink" href="#4-what-is-the-benefit-of-preparing-a-phd-and-spending-most-of-your-time-in-a-private-company-vs-a-university-lab">4. What is the benefit of preparing a PhD and spending most of your time in a private company vs. a University lab?</a></h2>
<p>Both environments are full of brilliant and highly-skilled people.
This is key for a young and curious Ph.D. candidate.
Many engineers at <span class="caps">NL</span> have a Ph.D. diploma, or at least they are very keen on research-oriented topics.
This culture of research is ideal to bridge between fundamental sciences and industrial applications.</p>
<p>Transferring scientific breakthroughs to industrial applications is an important challenge.
In the context of my Ph.D. at <span class="caps">NL</span>, I look forward to developing scientific understanding in consideration of industrial needs.
In particular, we expect the results of my Ph.D. to be immediately applicable to implementing Tezos.</p>
<h2 id="5-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs"><a class="toclink" href="#5-why-did-you-choose-to-become-a-part-of-the-blockchain-ecosystem-and-nomadic-labs">5. Why did you choose to become a part of the blockchain ecosystem and Nomadic labs?</a></h2>
<p>I enjoy programming and I enjoy trying out new ideas and seeing them be implemented in a running system!
I guess I’m a child of my generation.</p>
<p>Furthermore, direct experimentation is my preferred setup.
Often, scientists purposely abstract away technical details.
This makes research easier, but then integrating the results back in the real world can be complicated because many technical details remain unclear.</p>
<p><span class="caps">NL</span> embodies the integration of modern computer science along with the best technologies: a philosophy which I find inspiring and want to be a part of.</p>
<h1 id="questions-for-colins-nomadic-labs-mentor-yann-regis-gianas"><a class="toclink" href="#questions-for-colins-nomadic-labs-mentor-yann-regis-gianas">Questions for Colin’s Nomadic Labs mentor Yann Regis-Gianas</a></h1>
<h2 id="what-is-your-input-in-the-thesis-of-colin"><a class="toclink" href="#what-is-your-input-in-the-thesis-of-colin">What is your input in the thesis of Colin?</a></h2>
<p>As a mentor, I try to provide two kinds of input: <em>methodological advice</em> and <em>scientific expertise</em>.</p>
<p>Doing a Ph.D. is learning how to solve complex problems that have not already been solved.
This is a new setting for students, who are usually required to work on problems with known solutions.
Thus, a Ph.D. student must learn to explore a world of uncertainty, based on a rigorous analysis of the problem in question, and must acquire a deep understanding of the state of the art in the domain.</p>
<p>That’s indeed the second kind of input I provide:
with my experience in formal methods and programming languages, I can synthesise the state of our knowledge, typically by having my student Colin read the relevant research papers and interact with the scientific community in the field.</p>
<h2 id="why-is-this-thesis-important"><a class="toclink" href="#why-is-this-thesis-important">Why is this thesis important?</a></h2>
<p>Believe it or not, spreadsheets are the most widely-used programming language!</p>
<p>In <span class="caps">VLHCC</span>’05 <a href="https://doi.org/10.1109/VLHCC.2005.34">Christopher Scaffidi and his coauthors published <em>Estimating the number of end users and user programmers</em></a>, in which they concluded that there would be about 55 million spreadsheets users in the United States of America in 2012, compared to just 13 million professional programmers.
Many people with no background in programming can routinely define sophisticated computations using <span class="caps">MS</span>/Excel.
Many people use spreadsheets to maintain accounts, or even to automate complex accounting processes.</p>
<p>So: what if these spreadsheets could be turned into smart contracts and deployed on the Tezos blockchain?</p>
<p>Not only would this make programming smart contracts easier, but also spreadsheets are a good way to visualize the state of the contract once it is deployed on the chain: such a tool could help democratize blockchains.
Perhaps one day we might see people including something like this in their daily workflow!</p>
<p>If spreadsheets are to be used as smart contracts, then we need to make sure their definitions are correct.
Unfortunately, spreadsheets can be error-prone, and <a href="http://web.archive.org/web/20210510144002/https://www.teampay.co/insights/biggest-excel-mistakes-of-all-time/">famous errors in spreadsheets have turned into losses of millions of dollars</a>.
Colin’s Ph.D. will provide reasoning tools to help in the simulation and verification of smart contracts, giving mathematically-backed guarantees of correctness before contracts are deployed on the chain (something that should be standard for any smart contract, in fact!).</p>
<p>With his new vision about what <em>structured</em> spreadsheet programming is, Colin might even have an impact on the quality of the spreadsheets written by the spreadsheet industry of the future.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:affiliation">
<p>My research affiliation is: Université de Paris, <span class="caps">IRIF</span>, <span class="caps">INRIA</span>, Équipe-Project π.r2. But as a student, my enrolment is at the Université. <a class="footnote-backref" href="#fnref:affiliation" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Faster finality with Emmy*2021-05-03T12:00:00+02:002021-05-03T12:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-05-03:/faster-finality-with-emmy.html<p>Announcing forthcoming upgrade of consensus protocol, giving quicker time to finality (i.e. shorter transaction settlement times)</p><p>We are happy to announce that Emmy* is set to be included in the next Tezos protocol proposal Granada,<sup id="fnref:proposals"><a class="footnote-ref" href="#fn:proposals">1</a></sup> replacing the current consensus algorithm Emmy<sup>+</sup>.
If Granada is adopted, Emmy* will generally halve the time between blocks, from 60 seconds to 30 seconds, and allow transactions to achieve significantly faster finality than under the current consensus algorithm, Emmy<sup>+</sup>.</p>
<p>Specifically, Emmy* updates Emmy<sup>+</sup> by:</p>
<ul>
<li>a tweak in the definition of the minimal delay function, and</li>
<li>an increase in the number of endorsement slots per block</li>
</ul>
<p>both of which bring faster times to finality (in other words: shorter transaction settlement times) without compromising
security.<sup id="fnref:fin"><a class="footnote-ref" href="#fn:fin">3</a></sup> Please see <a href="https://gitlab.com/tzip/tzip/-/blob/master/drafts/current/draft_emmy-star.md">the
<span class="caps">TZIP</span></a>
for the specification and <a href="https://gitlab.com/tezos/tezos/-/issues/1027">issue
1027</a> for the implementation.</p>
<p>Thanks to these changes in Emmy*:</p>
<ul>
<li>a block can be produced with a delay of 30 seconds with respect to the
previous block if it has priority 0 and at least 60 percent of the total
endorsing power per block;</li>
<li>the number of endorsement slots per block is increased from 32 to 256.</li>
</ul>
<p>With these changes, on a healthy<sup id="fnref:healthy"><a class="footnote-ref" href="#fn:healthy">4</a></sup> chain and for a Byzantine attacker
with up to 33% stake for instance, the number of confirmations<sup id="fnref:confirm"><a class="footnote-ref" href="#fn:confirm">2</a></sup> decreases to 2
blocks, therefore 1 minute, a sixfold improvement.</p>
<p>The following plot shows the number of confirmations (in log scale) for
Emmy* vs Emmy<sup>+</sup> when varying the stake fraction from
0.1 to 0.4. This plot assumes the <a href="https://research-development.nomadic-labs.com/analysis-of-emmy.html#forks-started-in-the-past">“forks started in the past”
scenario</a>,
meaning that we are interested in the finality of a block which already has a
number of confirmations on top of it (and therefore, importantly, we know how
healthy the chain was in the meanwhile), and we ask ourselves whether this
number is sufficient. Here we assume a perfectly healthy chain. In the plot, the
highest red point corresponds to 12 confirmations.</p>
<p><img alt="Image" src="images/emmystar_announce/emmystar_vs_plus_past.png"></p>
<p>To complement the above plot, the following table presents a subset of the data
in text form. Each value in the table gives the number of confirmations for a given attacker stake fraction for
Emmy* and Emmy<sup>+</sup> in a “forks started in the past” scenario.</p>
<table>
<thead>
<tr>
<th align="center">Attacker stake</th>
<th align="center">Emmy*</th>
<th align="center">Emmy<sup>+</sup></th>
</tr>
</thead>
<tbody>
<tr>
<td align="center">0.1</td>
<td align="center">2</td>
<td align="center">2</td>
</tr>
<tr>
<td align="center">0.15</td>
<td align="center">2</td>
<td align="center">3</td>
</tr>
<tr>
<td align="center">0.2</td>
<td align="center">2</td>
<td align="center">3</td>
</tr>
<tr>
<td align="center">0.25</td>
<td align="center">2</td>
<td align="center">4</td>
</tr>
<tr>
<td align="center">0.3</td>
<td align="center">2</td>
<td align="center">5</td>
</tr>
<tr>
<td align="center">0.33</td>
<td align="center">2</td>
<td align="center">6</td>
</tr>
<tr>
<td align="center">0.35</td>
<td align="center">3</td>
<td align="center">7</td>
</tr>
<tr>
<td align="center">0.4</td>
<td align="center">3</td>
<td align="center">12</td>
</tr>
</tbody>
</table>
<p>So the final line above expresses that even if an attacker controls 40% of the stake, in Emmy* we only have to wait three blocks (90 seconds, assuming a good network, because the block time is 30 seconds) to be reasonably sure that our transaction is final, and in Emmy<sup>+</sup> we have to wait twelve blocks (12 minutes, because the block time is 60 seconds).</p>
<h3 id="transaction-finality-in-tezos-indexer"><a class="toclink" href="#transaction-finality-in-tezos-indexer">Transaction finality in tezos-indexer</a></h3>
<p>We are also happy to announce that
<a href="https://gitlab.com/nomadic-labs/tezos-indexer/">tezos-indexer</a> has <a href="https://gitlab.com/nomadic-labs/tezos-indexer/-/blob/master/src/db-schema/block_finality.sql">a new
feature</a>:
now it can tell you if your transaction is final in Emmy<sup>+</sup>.<sup id="fnref:star"><a class="footnote-ref" href="#fn:star">6</a></sup> For
instance: suppose your transaction is included in a block and that the next
three blocks get baked in 190 seconds; then the indexer will tell you that your
transaction is final with respect to:</p>
<ul>
<li>the security threshold in Emmy<sup>+</sup> which is <code>1e-8</code> and<sup id="fnref:plus"><a class="footnote-ref" href="#fn:plus">5</a></sup></li>
<li>an adversary with at most 20% stake.</li>
</ul>
<p>Note that 190 is just a bit more than 3 * 60 seconds, which is the minimal time needed to bake three blocks. </p>
<p>Now instead suppose that the three blocks get baked in more than 227 seconds: then
you will need to check again after one more block/confirmation. Say the fourth
block is baked in 60 seconds, so given that 4 blocks have been baked in about
287 seconds, the indexer will tell you that now you can consider your
transaction as final.</p>
<p>We are proud to share <a href="https://nomadic-labs.gitlab.io/emmyplus-experiments">a web demo</a>
for you to experiment for yourself with computing fork probabilities in both Emmy<sup>+</sup> and Emmy*.
It is based on <a href="https://research-development.nomadic-labs.com/emmy-in-the-partial-synchrony-model.html">our previous analysis of Emmy</a>.
If you want to dig deeper and are familiar with OCaml and Jupyter notebooks, you are welcome to
follow <a href="https://gitlab.com/nomadic-labs/emmyplus-experiments/-/tree/emmystar_exp">these instructions</a>
and let your creativity run free.</p>
<p><a name="disclaimer"></a></p>
<p><em>Please note <a href="https://research-development.nomadic-labs.com/analysis-of-emmy.html">our previous
disclaimer</a>: <a href="https://research-development.nomadic-labs.com/emmy-in-the-partial-synchrony-model.html">Our
analysis</a>
is based on a certain model of the blockchain and of an attacker. In
particular, the model assumes reasonable bounds on the communication delays:
that blocks and operations are diffused within 30 seconds. The attacker model
is that of a baker not following the rules of the protocol: it may double bake
and double endorse as it finds best in order to fork the chain. The attacker
model of our analysis </em><em>does not include</em><em> a more powerful attacker capable of
disturbing the network by blocking or delaying messages between honest participants.</em></p>
<p>Happy baking!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:proposals">
<p>The Tezos mainnet is currently running <a href="http://doc.tzalpha.net/protocols/008_edo.html">the Edo protocol</a>.
<a href="http://doc.tzalpha.net/protocols/009_florence.html">The Florence protocol</a> is currently <a href="https://www.tezosagora.org/proposal/11">being voted on</a>.
Assuming all goes well and the Florence upgrade is accepted, the Granada proposal will follow, and Emmy* should be part of that.
Subject to this all going well — we add this qualification because this is not something that Nomadic Labs can fully control; the voting process is up to you, our dear Tezos community! — then a <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">further upgrade from Emmy* to Tenderbake</a> is planned for a subsequent H proposal (name not yet fixed). <a class="footnote-backref" href="#fnref:proposals" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:confirm">
<p>A <strong>confirmation</strong> refers to a block on top of the block with a
transaction of interest to the user. <a class="footnote-backref" href="#fnref:confirm" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:fin">
<p>Emmy* is a Nakamoto-style algorithm (like Bitcoin), which means that its finality is <em>probabilistic</em>. Probabilistic finality means that as time passes we can become “reasonably sure” that a transaction we made is indeed included in the final blockchain (and not on some fork that might die out), where <em>reasonably sure</em> means “with probability
of being wrong smaller than some reasonable threshold” — which we quantify as <code>5 * 1e-9</code> (five in a trillion). This
puts our expectation of being wrong about a block being final at roughly once
every two centuries. <a class="footnote-backref" href="#fnref:fin" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:healthy">
<p>We call a chain <strong>healthy</strong> over a period of time when in this period
blocks have priority 0 and (almost) all endorsement slots are filled. A
concrete healthiness measure is the delay of the chain with respect to the ideal
chain where each block has a delay of one minute with respect to the previous
block. <a class="footnote-backref" href="#fnref:healthy" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:plus">
<p>In Emmy<sup>+</sup> blocks are baked at about 60
seconds, <code>1e-8</code> puts our expectation of being wrong about a block being final
at the same roughly once every two centuries, which corresponds to the security
threshold of <code>5 * 1e-9</code> in Emmy*.
That is: we use a more rigorous security threshold in Emmy* because it’s faster so we can bake more blocks every two centuries. <a class="footnote-backref" href="#fnref:plus" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:star">
<p>This new feature of tezos-indexer will be updated with respect to
Emmy* once Emmy* is adopted. <a class="footnote-backref" href="#fnref:star" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
</ol>
</div>Announcing the report “Possible evolutions of the voting system in Tezos”2021-04-14T14:00:00+02:002021-04-14T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-04-14:/announcing-the-report-possible-evolutions-of-the-voting-system-in-tezos.html<p>Announcing a report on possible evolutions of the voting system in Tezos</p><p>Nomadic labs has an <a href="https://www.inria.fr/en/inria-and-nomadic-labs-join-forces-develop-research-programme-blockchain-technology">ongoing research relationship</a> with <a href="https://www.inria.fr/en"><span class="caps">INRIA</span></a> (a French national technology research agency).</p>
<p>In the context of this relationship, Nomadic Labs commissioned a short report to explore what a privacy-preserving amendment procedure might look like on Tezos, authored by three experts in voting protocols and cryptography: <a href="https://members.loria.fr/VCortier/">Véronique Cortier</a>, <a href="https://members.loria.fr/PGaudry/">Pierrick Gaudry</a> and <a href="https://stephane.glondu.net">Stéphane Glondu</a>.</p>
<p>There is no plan to implement the contents of the report for now, but we welcome and encourage feedback on its findings: from regular Tezos users, and from research and industry experts.</p>
<p><a href="https://research-development.nomadic-labs.com/files/2021-01-report-private-amendment.pdf">The report is here</a> and we encourage you to leave comments in this <a href="https://forum.tezosagora.org/t/announcing-the-report-possible-evolutions-of-the-voting-system-in-tezos/3002">agora post</a>. Thank you.</p>Meanwhile at Nomadic Labs #112021-04-09T10:00:00+02:002021-04-09T10:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-04-09:/meanwhile-at-nomadic-labs-11.html<p>A summary of Nomadic Labs activities in the first quarter of 2021</p><p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>Welcome to our <a href="https://research-development.nomadic-labs.com/blog.html?category=meanwhile">meanwhile series</a>, the ongoing story of Nomadic Labs’ amazing adventures in the Tezos blockchain space.
This post is a recap of our activities in the first quarter of 2021, following on from our <a href="https://research-development.nomadic-labs.com/a-review-of-nomadic-labs-in-2020.html">2020 recap</a>.
As always, you can find out more about us here:</p>
<blockquote>
<p><a href="https://twitter.com/labosnomades?lang=en">Twitter @LabosNomades</a> ~
<a href="https://www.nomadic-labs.com">Website</a> ~
<a href="https://fr.linkedin.com/company/nomadic-labs">LinkedIn</a> ~
<a href="https://research-development.nomadic-labs.com/blog.html">Technical blog</a> ~
<a href="https://gitlab.com/nomadic-labs">GitLab repo</a> </p>
</blockquote>
<p>So here’s what we’ve been up to these past three months:</p>
<ul>
<li><a href="#edo-protocol">Edo protocol upgrade and Florence protocol proposal</a></li>
<li><a href="#easier--safer--better-installation-of-the-tezos-codebase">Easier, safer, better installation of the Tezos codebase</a></li>
<li><a href="#culture-and-growth">Culture and growth</a></li>
<li><a href="#announcing-1-000-days-of-tezos-mainnet">Announcing 1,000 days of Tezos mainnet</a></li>
<li><a href="#adoption-and-support">Adoption and support</a></li>
<li><a href="#dexter">Dexter</a></li>
<li><a href="#umami">Umami</a></li>
<li><a href="#forthcoming-switch">Forthcoming switch from Emmy<sup>+</sup> to Emmy<sup>★</sup> and then Tenderbake</a></li>
<li><a href="#nl-research-seminars"><span class="caps">NL</span> research seminars</a></li>
<li><a href="#research-and-sponsorship">Research and sponsorship</a></li>
<li><a href="#prochaine">À la prochaine</a></li>
</ul>
<p><a name="edo-protocol"></a></p>
<h2 id="edo-protocol-upgrade-and-florence-protocol-proposal"><a class="toclink" href="#edo-protocol-upgrade-and-florence-protocol-proposal">Edo protocol upgrade and Florence protocol proposal</a></h2>
<p>The <a href="https://tezos.com">Tezos blockchain</a> contains a <a href="https://tezos.com/docs/learn/what-is-tezos#self-amendment">self-amendment mechanism to upgrade the protocol</a> by community vote, meaning: no need for hard forks, and in-built agility and ability to adopt new ideas in the fast-evolving blockchain space.
We intend <a href="https://forum.tezosagora.org/t/regular-scheduling-for-our-tezos-proposals/2331">to follow a regular protocol upgrade schedule</a>.</p>
<p>The recent <a href="https://forum.tezosagora.org/t/announcing-the-edo-release/2443">Edo protocol upgrade</a> on <a href="https://tzstats.com/election/25">13 February 2020</a> went smoothly (block height 1,398,551; cycle 341; <a href="https://tezos.gitlab.io/protocols/008_edo.html">changelog</a>).<sup id="fnref:delphi"><a class="footnote-ref" href="#fn:delphi">1</a></sup>
Edo introduced several substantive new features to Tezos:</p>
<ul>
<li><a href="https://research-development.nomadic-labs.com/sapling-integration-in-tezos-tech-preview.html">Zcash Sapling integration</a> (privacy-preserving transactions; see <a href="https://tezos.gitlab.io/008/sapling.html#sapling-dev-008">doc</a> and <a href="https://decrypt.co/49912/tezos-will-get-zcashs-privacy-features-in-next-upgrade">an accessible explanation</a>). </li>
<li><a href="https://tezos.gitlab.io/008/michelson.html#michelsontickets-008">Tickets</a> (mechanism for smart contracts to authenticate data with respect to a Tezos address).</li>
<li><a href="https://tezos.gitlab.io/protocols/008_edo.html#voting-procedure">An extended Voting procedure</a> (adds a two-week “Adoption” phase).</li>
<li><a href="https://tezos.gitlab.io/protocols/008_edo.html">… and more</a>. See also a <a href="https://medium.com/tezoscommons/tezos-upgrade-analysis-edo-2-0-4e0df2eb0f77">detailed analysis of the Edo upgrade</a>.</li>
</ul>
<p>Associated to the Edo protocol upgrade is also a new <a href="http://tezos.gitlab.io/releases/version-8.html">major release</a> of <code>tezos-node</code> that contains an update to the protocol environment<sup id="fnref:protocol-environment"><a class="footnote-ref" href="#fn:protocol-environment">2</a></sup>, numbered <a href="http://tezos.gitlab.io/api/odoc/_html/tezos-protocol-environment/Tezos_protocol_environment/module-type-V1/index.html">“Version 1”</a>.
This is significant because until now, all protocols have used <a href="http://tezos.gitlab.io/api/odoc/_html/tezos-protocol-environment/Tezos_protocol_environment/module-type-V0/index.html">“Version 0”</a>.</p>
<p>Together with
<a href="https://gitlab.com/marigold">Marigold</a>,
<a href="https://tarides.com">Tarides</a>,
<a href="https://www.dailambda.jp/">DaiLambda</a>, and
<a href="http://keefertaylor.com">Keefer Taylor</a>,
we proposed <a href="https://research-development.nomadic-labs.com/florence-our-next-protocol-upgrade-proposal.html">a successor protocol upgrade, <strong>Florence</strong></a>. See</p>
<ul>
<li><a href="https://tezos.gitlab.io/protocols/009_florence.html">the detailed Florence changelog</a>,</li>
<li><a href="https://research-development.nomadic-labs.com/florence-our-next-protocol-upgrade-proposal.html">an accessible discussion</a>. </li>
</ul>
<p>One standout feature of Florence is <a href="https://research-development.nomadic-labs.com/sound-and-fast-gas-monitoring-with-saturation-arithmetic.html">a new arithmetic system (based on <em>saturation arithmetic</em>) for computing gas costs in Michelson smart contracts</a>.
Our benchmarks indicate a <a href="https://research-development.nomadic-labs.com/sound-and-fast-gas-monitoring-with-saturation-arithmetic.html#what-are-the-gains">tenfold speedup of gas computation, and a 35% speedup of the execution cycle of the smart contract Michelson interpreter in Florence overall</a>.<sup id="fnref:execution-cycle"><a class="footnote-ref" href="#fn:execution-cycle">3</a></sup>
So smart contracts in Florence can be smarter, for the same gas.<sup id="fnref:execution-cycle-2"><a class="footnote-ref" href="#fn:execution-cycle-2">4</a></sup></p>
<p>Another notable feature is <a href="https://research-development.nomadic-labs.com/tezos-calling-convention-migrating-from-breadth-first-to-depth-first-order-bfs-to-dfs.html">a migration from <span class="caps">BFS</span> to <span class="caps">DFS</span></a>.
See also:</p>
<ul>
<li>The <a href="https://gitlab.com/tzip/tzip/-/merge_requests/111"><span class="caps">DFS</span> Tezos Improvement Proposal (<span class="caps">TZIP</span>)</a>.</li>
<li>A <a href="https://gitlab.com/tezos/tezos/-/merge_requests/2420">detailed discussion of the code design</a>.</li>
</ul>
<p>The main developer of this feature is <a href="https://gitlab.com/rueshyna">ChiaChi Tsai</a> from
<a href="https://gitlab.com/marigold">Marigold</a>.
Nomadic Labs’ participation in the <span class="caps">BFS</span> to <span class="caps">DFS</span> migration was in the design (especially security considerations), to review several versions of the code, and to perform a major replay test.<sup id="fnref:replay"><a class="footnote-ref" href="#fn:replay">5</a></sup></p>
<h2 id="easier-safer-better-installation-of-the-tezos-codebase"><a class="toclink" href="#easier-safer-better-installation-of-the-tezos-codebase">Easier, safer, better installation of the Tezos codebase</a></h2>
<p>The <a href="https://tezos.gitlab.io/introduction/howtoget.html">“How to get Tezos” installation instructions</a> are now automatically checked as part of the build process itself, meaning they are automatically tested and included in documentation and should stay correct and up-to-date <em>even as they depend</em> on an extensive environment in constant evolution, due to</p>
<ul>
<li>development of Tezos itself (e.g. switching to new protocols, releasing new versions of the platform, adding dependencies), and </li>
<li>evolution of third-party software packages (e.g. the <span class="caps">OPAM</span> package manager for OCaml, individual OCaml packages, Linux releases, and so forth). </li>
</ul>
<p>Technically, this was accomplished by writing a library of executable scripts for each installation scenario (binaries, docker images, compiling from source, …) which are executed in the <span class="caps">CI</span> (continuous integration) process itself, and automatically copied over each time documentation is generated and published.<sup id="fnref:contrast"><a class="footnote-ref" href="#fn:contrast">6</a></sup></p>
<p>This new system means </p>
<ul>
<li>early and automatic detection and quick correction of installation problems, and </li>
<li>no more copy/paste errors or stale install scripts in documentation </li>
</ul>
<p>— thus making installation more reliable and convenient for all Tezos users, from neophytes to experienced developers.</p>
<p>In summary: the Tezos installation scripts and relevant documentation are now part of the testable codebase of Tezos itself.</p>
<h2 id="culture-and-growth"><a class="toclink" href="#culture-and-growth">Culture and growth</a></h2>
<p>Since January 2021 we are delighted to have been joined by three new hires and three interns, bringing our count of full-time employees to 62. </p>
<h2 id="announcing-1000-days-of-tezos-mainnet"><a class="toclink" href="#announcing-1000-days-of-tezos-mainnet">Announcing 1,000 days of Tezos mainnet</a></h2>
<p>As of Friday 26 March 2021, the Tezos mainnet is 1,000 days old.
See this subtle poster on Reddit:</p>
<blockquote class="reddit-card" data-card-created="1616995712"><a href="https://www.reddit.com/r/tezos/comments/mdul0w/1000_days_of_tezos_mainnet/">1000 Days of Tezos Mainnet</a> from <a href="http://www.reddit.com/r/tezos">r/tezos</a></blockquote>
<script async src="//embed.redditmedia.com/widgets/platform.js" charset="UTF-8"></script>
<p>Specifically, the chain was launched on June 30 2018, as per the timestamp of <a href="https://tzstats.com/0">the genesis Tezos block</a>.<sup id="fnref:zero"><a class="footnote-ref" href="#fn:zero">7</a></sup></p>
<p>(The main Tezos network wasn’t called “Tezos Mainnet” at the time but “Tezos Betanet”.
It was renamed <a href="https://www.coindesk.com/no-longer-beta-200-million-tezos-blockchain-will-be-official-mainnet-soon">on September 17</a>, once devs were confident in its stability.
<a href="https://en.wikipedia.org/wiki/Prince_(musician)">Following the precedent of a famous singer</a>, we might say that “The blockchain formerly known as Tezos Betanet” is now 1,000 days old.)</p>
<p>You can check the arithmetic in a Unix system by running</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>date<span class="w"> </span>-d<span class="w"> </span><span class="s2">"June 30 2018 + 1000 days"</span>
Fri<span class="w"> </span><span class="m">26</span><span class="w"> </span>Mar<span class="w"> </span><span class="m">00</span>:00:00<span class="w"> </span>GMT<span class="w"> </span><span class="m">2021</span>
</code></pre></div>
<h2 id="adoption-and-support"><a class="toclink" href="#adoption-and-support">Adoption and support</a></h2>
<ul>
<li>
<p>Nomadic Labs has joined <a href="https://infrachain.com">Infrachain</a>, a Luxembourg-based European cross-industry effort to pool blockchain expertise and promote regulatory-compliant adoption.
See <a href="https://infrachain.com/new-members-compellio-and-nomadic-labs-join-infrachain/">the announcement</a> (in English) and <a href="https://paperjam.lu/article/nomadic-labs-bonne-pioche-infr">a short article in Paperjam</a> (in French).</p>
</li>
<li>
<p>We are pleased to welcome two new institutional bakers on the Tezos chain:</p>
<ul>
<li><a href="https://www.wakam.com/en/">Wakam</a> (a digital insurance and bespoke insurance creator) <a href="https://www.wakam.com/en/tezos-baker/">is now a Tezos baker</a> (see <a href="https://www.wakam.com/en/wakam-joins-the-tezos-ecosystem-as-corporate-baker/">the press release</a>; also as <a href="https://nomadic-labs.com/download/PR-Wakam-Baker.pdf">a pdf</a>).</li>
<li><a href="https://www.theblockchain-group.com">The Blockchain Group</a> (a blockchain service and technology group), through its subsidiary <a href="https://www.theblockchainxdev.com">The Blockchain Xdev</a>, <a href="https://www.theblockchainxdev.com/the-blockchain-xdev-rejoint-lecosysteme-tezos-comme-baker-institutionnel/">is now a Tezos baker</a> (see <a href="https://nomadic-labs.com/download/PR-TBG-Baker.pdf">the press release</a>). </li>
</ul>
</li>
<li>
<p><a href="https://www.linkedin.com/feed/update/urn:li:activity:6777997701989584896">Nomadic Labs has helped</a> to deploy <a href="https://www.lugh.io">Lugh</a> (currency name <span class="caps">EURL</span>).<sup id="fnref:lugh"><a class="footnote-ref" href="#fn:lugh">8</a></sup>
Lugh is a Euro-backed <a href="https://en.wikipedia.org/wiki/Digital_asset"><strong>digital asset</strong></a>, meaning that 1 <span class="caps">EURL</span> is always equal to 1€ (see <a href="https://c621f0ab-b4a0-458b-a736-34d403489987.filesusr.com/ugd/5bc33e_f0cb480bb7284599a66b02f5f7599305.pdf">press release</a>).
If you hold 1 <span class="caps">EURL</span> then this corresponds to 1€ in the balance of a certain bank account at Société Générale.
Lugh has been <a href="https://c621f0ab-b4a0-458b-a736-34d403489987.filesusr.com/ugd/5bc33e_8ce5d98d895745369d829f589cc70266.pdf">reviewed by a “big four” firm (PriceWaterhouseCoopers)</a>.
See also <a href="https://twitter.com/i/status/1372486868781182977">this short video introducing Lugh</a>, and <a href="https://www.linkedin.com/company/lughio/">the Lugh LinkedIn page</a>.</p>
</li>
</ul>
<h2 id="dexter"><a class="toclink" href="#dexter">Dexter</a></h2>
<p><a href="https://dexter.exchange">Dexter</a> is a smart contract to enable trade between tez and any <a href="https://assets.tqtezos.com/docs/token-contracts/fa12/1-fa12-intro/"><span class="caps">FA1</span>.2-compliant</a> token (<a href="https://better-call.dev/dapps/dexter">dapp</a>; <a href="https://dexter.exchange/docs/dexter-intro/">doc</a>; <a href="https://dexter.exchange/docs/dexter-tutorials/">tutorials</a>).
Think: “<a href="https://uniswap.org/">Uniswap</a> for Tezos”.</p>
<p>Examples of smart contract implementations satisfying the <span class="caps">FA1</span>.2 interface include:</p>
<ol>
<li>The <a href="https://better-call.dev/dapps/ethtz"><span class="caps">ETH</span>-wrapped token ETHtz</a> (<a href="https://ethtz.io">ETHtz homepage</a>) </li>
<li>The <a href="https://better-call.dev/dapps/usdtz"><span class="caps">USD</span>-pegged stablecoin USDtz</a> (<a href="https://usdtz.com/">USDtz homepage</a>)</li>
<li>The <a href="https://better-call.dev/dapps/tzbtc">Wrapped Bitcoin tzBTC</a> (<a href="https://tzbtc.io/">tzBTC homepage</a>)</li>
</ol>
<p>Thus using Dexter, a user can exchange tez for ETHtz, USDtz, and tzBTC — all on the Tezos blockchain.<sup id="fnref:how-it-works"><a class="footnote-ref" href="#fn:how-it-works">9</a></sup></p>
<p>Nomadic Labs verified that the <a href="https://research-development.nomadic-labs.com/dexter-decentralized-exchange-for-tezos-formal-verification-work-by-nomadic-labs.html">Dexter implementation conforms to the Dexter specification</a> — this is called <strong>verifying the functional specification</strong> — using the <a href="https://coq.inria.fr/">Coq proof assistant</a> with the <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/">Mi-Cho-Coq framework</a>.
This was in addition to a <a href="https://github.com/trailofbits/publications/blob/master/reviews/dexter.pdf">security audit</a> by <a href="https://www.trailofbits.com/">Trail of Bits</a> and <a href="https://gitlab.com/camlcase-dev/dexter/-/tree/master/morley/tests">property-based testing</a> by camlCase. </p>
<p>However, we discovered in late February while working on the <a href="https://research-development.nomadic-labs.com/florence-our-next-protocol-upgrade-proposal.html">Florence upgrade proposal</a> that <a href="https://research-development.nomadic-labs.com/dexter-flaw-discovered-funds-are-safe.html">the Dexter specification itself contained an error</a>. </p>
<p>Just because something is a specification does not make it right: specifications can have mistakes just like anything else (indeed <a href="https://research-development.nomadic-labs.com/dexter-decentralized-exchange-for-tezos-formal-verification-work-by-nomadic-labs.html#future-work">we commented on this in the initial blog post</a>).
This is why responsible ecosystems like Tezos subject complex systems like Dexter to <em>multiple overlapping verification efforts</em>.<sup id="fnref:quick-ship"><a class="footnote-ref" href="#fn:quick-ship">10</a></sup>
So this is just another day in The Real World.</p>
<p>We reported the error and a “White Knight” operation was used to remove funds from the contract — using the error itself, by the way — and to return funds to their owners.</p>
<p>A rewrite of Dexter is under development in a collaboration between Nomadic Labs and <span class="caps">LIGO</span> (see <a href="https://gitlab.com/dexter2tz/dexter2tz/">this GitLab repo</a>).
We at Nomadic Labs are carrying out <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/merge_requests/100">a formal verification</a> of it.
So far, we’ve finished the verification of a functional specification, which brings confidence up to the level of the previous version of Dexter.
Obviously we should go further, and the way to do this is to verify high-level properties of the new specification, to check that no further errors exist like the one discovered in the previous version. </p>
<p>In the meantime, and to avoid delays for the existing users of Dexter, the Dexter contract has been patched (<a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/merge_requests/98">patch source code</a>) to fix the flaw, and <a href="https://camlcase.io/blog/dexter-is-back/">brought back online</a>. </p>
<h2 id="umami"><a class="toclink" href="#umami">Umami</a></h2>
<p>Our Wallet team has been diligently at work on the new <a href="https://umamiwallet.com">Umami wallet</a>.
Umami is a <a href="https://en.wikipedia.org/wiki/Cryptocurrency_wallet">cryptocurrency wallet</a> for Tezos giving both beginner users and power users convenient access to all features available within the Tezos protocol including: multiple accounts, tokens, batch transactions, and delegation — and eventually, though not in the initial beta: contracts and use of hardware wallets.</p>
<p>From April 20th you will be able to <a href="https://umamiwallet.com/#download">download Umami beta here</a>.
See also the <a href="https://gitlab.com/nomadic-labs/umami-wallet/umami">Umami GitLab repo</a>. </p>
<p><a name="forthcoming-switch"></a></p>
<h2 id="forthcoming-switch-from-emmy-to-emmy-and-then-tenderbake"><a class="toclink" href="#forthcoming-switch-from-emmy-to-emmy-and-then-tenderbake">Forthcoming switch from Emmy<sup>+</sup> to Emmy<sup>★</sup> and then Tenderbake</a></h2>
<p>We plan to upgrade our consensus algorithm from the current Emmy<sup>+</sup> to Emmy<sup>★</sup>, and shortly thereafter to Tenderbake. </p>
<p>Emmy<sup>+</sup> is a Nakamoto-style consensus algorithm.
This means that it is similar to the Bitcoin consensus algorithm, just adapted to Tezos’ proof-of-stake system rather than the proof-of-work system used by Bitcoin.
Emmy<sup>★</sup> refines the Emmy<sup>+</sup> system in two ways (listed in decreasing order of significance):</p>
<ol>
<li>Emmy<sup>★</sup> provides a special fast consensus path for when the network is operating normally, which is most of the time. This makes Emmy<sup>★</sup> faster than Emmy<sup>+</sup> when things are going well — and no slower when they’re not.</li>
<li>Emmy<sup>★</sup> increases the number of endorsement slots from 32 to 256. This is a technical tweak designed to increase stability and participation.</li>
</ol>
<p>How does Emmy<sup>★</sup> fare at scale in a test that approximates a real-live system?
This is what <strong>resillience testing</strong> is about.
<a href="https://tqtezos.com">TQTezos</a>, with support from Nomadic Labs, has constructed a resilience test network framework based on Kubernetes which allows us to quickly deploy a large testnet on <span class="caps">AWS</span>.
It takes about 30 minutes to deploy a 400 node private network and start baking on it.<sup id="fnref:resilience"><a class="footnote-ref" href="#fn:resilience">11</a></sup></p>
<p>After the upgrade to Emmy<sup>★</sup> there will follow an upgrade to <strong>Tenderbake</strong>.
Tenderbake belongs to the classical <span class="caps">BFT</span>-style family, so the switch from the Nakamoto-style to the classical <span class="caps">BFT</span>-style is a big deal.
We discuss this in detail in <a href="https://research-development.nomadic-labs.com/a-look-ahead-to-tenderbake.html">a blog post looking ahead to Tenderbake</a>.
In summary:</p>
<ol>
<li>Tenderbake offers <em>deterministic finality</em>, with finality time of one minute assuming good network behaviour. This means that (assuming the network is stable and not under attack) a block becomes final in one minute (i.e. the transaction settlement time is one minute).</li>
<li>Tenderbake has no forks. If the network degrades, the Tenderbake consensus algorithm waits until connectivity is restored.<sup id="fnref:safe"><a class="footnote-ref" href="#fn:safe">12</a></sup></li>
</ol>
<h2 id="nl-research-seminars"><a class="toclink" href="#nl-research-seminars"><span class="caps">NL</span> research seminars</a></h2>
<p>Our series of <a href="https://www.youtube.com/watch?v=2jd-1VxzmQ0&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7">Nomadic Labs research seminars</a> has continued at a regular pace:</p>
<ol>
<li><a href="https://www.youtube.com/watch?v=2jd-1VxzmQ0&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=1">Practical proofs using Juvix</a> (27 October 2020)</li>
<li><a href="https://www.youtube.com/watch?v=SwD7fpSNnis&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=2">Verifying smart contracts using Mi-Cho-Coq</a> (10 November 2020)</li>
<li><a href="https://www.youtube.com/watch?v=Aha0OkKkw5A&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=3">Efficient data storage on the blockchain using Plebeia</a> (24 November 2020)</li>
<li><a href="https://www.youtube.com/watch?v=mel76DFerL0&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=4">Adding multicore programming to OCaml</a> (08 December 2020)</li>
<li><a href="https://www.youtube.com/watch?v=zIVR8gc3ofE&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=5">zkChannels in Tezos</a> (05 January 2021) </li>
<li><a href="https://www.youtube.com/watch?v=71by_7JcDdU&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=6">Towards mechanised verification of the <span class="caps">LIGO</span> compiler</a> (20 January 2021)</li>
<li><a href="https://www.youtube.com/watch?v=u1duZUptIcc&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=7">SmartPy: the inner workings</a> (02 February 2021)</li>
<li><a href="https://www.youtube.com/watch?v=Q1esIDbcaLU&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=8">Bringing K Powered Blockchain Security to Tezos</a> (16 February 2021)</li>
<li><a href="https://www.youtube.com/watch?v=_3Lwxr6AjgE&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=9">Implementing Checker, a Robocoin Mechanism for Tezos</a> (02 March 2021)</li>
<li><a href="https://www.youtube.com/watch?v=7oKa9mjnfZU&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=10">High-level smart contract design <span class="amp">&</span> verification with Archetype</a> (16 March 2021)</li>
</ol>
<h2 id="research-and-sponsorship"><a class="toclink" href="#research-and-sponsorship">Research and sponsorship</a></h2>
<p>The Tezos Foundation was a platinum sponsor of <span class="caps">POPL</span> 2021 in January 2021 (and of <span class="caps">POPL</span> 2020 and <span class="caps">POPL</span> 2019), and Nomadic Labs sponsored the colocated <a href="https://popl21.sigplan.org/home/CPP-2021"><span class="caps">CPP</span> 2021 (Conference on Certified Programs and Proofs)</a> (and <span class="caps">CPP</span> 2020).
Our engineers had a strong presence at these events, including: </p>
<ul>
<li>
<p>On Monday 18 Jan, Arvid Jakobsson presented <a href="https://www.youtube.com/watch?v=sFMJBTtbjTc&t=1048s">the formalisation of the Dexter <em>decentralised exchange</em> contract in the Mi-Cho-Coq framework</a> at <a href="https://popl21.sigplan.org/room/POPL-2021-venue-cpp"><span class="caps">CPP</span>’s 2021 Lightning Talks session</a><sup id="fnref:Dexter"><a class="footnote-ref" href="#fn:Dexter">13</a></sup>.</p>
</li>
<li>
<p>On Wednesday 20 Jan, Michel Mauny, Nomadic Labs’ <span class="caps">CEO</span>, made a short presentation of our company at <a href="https://popl21.sigplan.org/details/POPL-2021-research-papers/74/Sponsor-Reception"><span class="caps">POPL</span>’s Sponsor Reception</a>, which also featured a contributed video <a href="https://www.youtube.com/watch?v=muATX5DnvnQ">presenting a general overview of Tezos by Arthur Breitman</a>.</p>
</li>
<li>
<p>On Friday 22 Jan, <a name="fcsl-pres">Germán Delbianco</a> presented new developments in the <a href="https://app.clowdr.org/conference/popl2021/item/f35f227c-58c5-4fb3-8100-a3eebe37833c">algebraic foundations of Concurrent Separation logics</a> at <span class="caps">POPL</span><sup id="fnref:FCSL"><a class="footnote-ref" href="#fn:FCSL">14</a></sup>.</p>
</li>
</ul>
<p>You can read our <a href="https://research-development.nomadic-labs.com/popl-2021-retrospective.html">full <span class="caps">POPL</span> 2021 retrospective here</a>.</p>
<p><a name="prochaine"></a></p>
<h2 id="a-la-prochaine"><a class="toclink" href="#a-la-prochaine">À la prochaine</a></h2>
<p>So that’s it!
The first three months of 2021 have been eventful and productive, and the next three months surely will be too.
Thanks for reading, and do check in again next quarter for the next Meanwhile.</p>
<p><small><i><a href='http://ecotrust-canada.github.io/markdown-toc/'>Table of contents generated with markdown-toc</a></i></small></p>
<div class="footnote">
<hr>
<ol>
<li id="fn:delphi">
<p>The previous protocol upgrade was <a href="https://research-development.nomadic-labs.com/delphi-official-release.html">Delphi</a> on <a href="https://tzstats.com/election/24">12 November 2020</a> (block height 1,212,417; cycle 296; <a href="https://research-development.nomadic-labs.com/delphi-changelog.html">changelog</a>; <a href="https://research-development.nomadic-labs.com/smarter-contracts-thanks-to-delphi-part-12.html">significance of the upgrade</a>). <a class="footnote-backref" href="#fnref:delphi" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:protocol-environment">
<p>The <em>protocol environment</em> is a library of cryptographic primitives and other useful functions (packaged as <a href="http://tezos.gitlab.io/api/odoc/_html/tezos-protocol-environment/Tezos_protocol_environment/index.html">an OCaml module</a>). <a class="footnote-backref" href="#fnref:protocol-environment" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:execution-cycle">
<p>The term <strong>execution cycle</strong> in the context of gas costing is the gas cost of decoding instructions, dispatching them, and of gas monitoring itself. So ‘execution cycle’ does not include the cost of actually executing instructions; it is the cost of arranging for instructions to execute … <a class="footnote-backref" href="#fnref:execution-cycle" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:execution-cycle-2">
<p>… thus, a theoretical smart contract in Florence, consisting of instructions that do zero work and cost zero gas, can do this nothing up to 35% <em>faster</em> (and for <span class="math">\(1/1.35\)</span> of the gas) than it would in Edo! <a class="footnote-backref" href="#fnref:execution-cycle-2" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:replay">
<p>A <strong>replay test</strong> is when chain history is replayed using the newer protocol, to check for breaking changes and to check that the functionality of any existing live smart contracts is not affected. See also <a href="https://research-development.nomadic-labs.com/tezos-calling-convention-migrating-from-breadth-first-to-depth-first-order-bfs-to-dfs.html">the blog post</a>; search for “Replays of on-chain history”. <a class="footnote-backref" href="#fnref:replay" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:contrast">
<p>Contrast this with <em>paper-only</em> scripts, which exist in the documentation but must be manually tested and copy/pasted. <a class="footnote-backref" href="#fnref:contrast" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:zero">
<p>The <strong>genesis</strong> block of a blockchain is its first block. <a href="https://tzstats.com/0">It is numbered 0</a> (not 1) because programmers and computer scientists start counting at zero: thus zero is the first number, one is the second, and so on. <a class="footnote-backref" href="#fnref:zero" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn:lugh">
<p>Lugh is an <a href="https://en.wikipedia.org/wiki/Lugh">Irish deity</a> associated with craftsmanship and skill, roughly corresponding to Mercury in Roman mythology and Apollo in Greek mythology. <a class="footnote-backref" href="#fnref:lugh" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
<li id="fn:how-it-works">
<p>Here’s how it works: tez is the <em>primitive token</em> of the Tezos blockchain.
Tezos has no other primitive tokens.
However, Tezos does have smart contracts, which gives huge flexibility, and in particular ETHtz, USDtz, and tzBTC are <em>smart contracts</em> on the Tezos blockchain which implement ledgers giving the effect of what we will call <em>non-primitive tokens</em> in this footnote — e.g. they include ledger-like entrypoints such as <code>%transfer</code>, <code>%getBalance</code>, and <code>%getTotalSupply</code>.</p>
<p>The <span class="caps">FA1</span>.2 standard is a precise specification which smart contracts intending to implement non-primitive tokens, can satisfy.
Then Dexter is a smart contract which implements functionality to convert between tez and any non-primitive token, provided that the smart contract implementing the non-primitive token satisfies the <span class="caps">FA1</span>.2 specification. <a class="footnote-backref" href="#fnref:how-it-works" title="Jump back to footnote 9 in the text">↩</a></p>
</li>
<li id="fn:quick-ship">
<p>This means that yes, sometimes more work is required so that features ship later than we hoped. That’s not a bug, that’s a feature. And it’s not unique to blockchain. This is what it’s like to work with any complex system, be it a bank, an aircraft design, or getting your child into bed. <a class="footnote-backref" href="#fnref:quick-ship" title="Jump back to footnote 10 in the text">↩</a></p>
</li>
<li id="fn:resilience">
<p>Think: “easy test flights, for software”.
You can fire up a large test system and watch it run as you tweak parameters, play with network properties, and so check the real-world conditions in which the system performs well as an engineering structure.
The infrastructure toolchain is built on Helm, Kubernetes and Docker.
You can also use <em>Pulumi</em> to automate your deployment into <span class="caps">AWS</span> <span class="caps">EKS</span>.
For local deployments the toolchain utilises <em>minikube</em>, which enables comfortable set-up of a local network of up to 20 nodes (depending on available <span class="caps">RAM</span>).</p>
<p>The code is open-source and accessible via the <a href="https://github.com/tqtezos/tezos-k8s">tezos-k8s github repo</a>. It is in active development, and the developers welcome external contributions in bug fixes, feature code, and requirements definition. <a class="footnote-backref" href="#fnref:resilience" title="Jump back to footnote 11 in the text">↩</a></p>
</li>
<li id="fn:safe">
<p>Classical <span class="caps">BFT</span>-style algorithms are <em>safe</em>, but not <em>live</em>, whereas Nakamoto-style algorithms are <em>live</em> but not <em>safe</em>. This means that a classical <span class="caps">BFT</span>-style algorithm (like Tenderbake) will safely wait out periods of network degradation and continue once connectivity is restored; whereas a Nakamoto-style algorithm will remain live and fork during a period of network degradation, and the fork collapses once connectivity is restored. <a class="footnote-backref" href="#fnref:safe" title="Jump back to footnote 12 in the text">↩</a></p>
</li>
<li id="fn:Dexter">
<p>Arvid Jakobsson, Colin González, Bruno Bernardo, and Raphaël Cauderlier. <em>Formally Verified Decentralized Exchange with Mi-Cho-Coq</em>. Contributed Lightning Talk to <a href="https://popl21.sigplan.org/room/POPL-2021-venue-cpp"><span class="caps">CPP</span> 2021</a>. Thanks also to Kristina Sojakova (<span class="caps">INRIA</span>) and James Haver (camlCase) for their contributions to the formalisation effort. <a class="footnote-backref" href="#fnref:Dexter" title="Jump back to footnote 13 in the text">↩</a></p>
</li>
<li id="fn:FCSL">
<p>František Farka, Aleksandar Nanevski, Anindya Banerjee, Germán Andrés Delbianco, and Ignacio Fábregas. <em>On Algebraic Abstractions for Concurrent Separation Logics</em>. Proc. <span class="caps">ACM</span> Program. Lang. 5, <span class="caps">POPL</span>, Article 5 (January 2021), 32 pages. <a href="https://doi.org/10.1145/3434286">https://doi.org/10.1145/3434286</a> <a class="footnote-backref" href="#fnref:FCSL" title="Jump back to footnote 14 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Sound and fast gas monitoring with saturation arithmetic2021-04-02T13:00:00+02:002021-04-02T13:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2021-04-02:/sound-and-fast-gas-monitoring-with-saturation-arithmetic.html<p>Fast calculation of gas costs using saturation arithmetic. With speed comes some theoretical loss of safety, but in practice it works well.</p><h1 id="sound-and-fast-gas-monitoring-lets-use-saturation-arithmetic"><a class="toclink" href="#sound-and-fast-gas-monitoring-lets-use-saturation-arithmetic">Sound and fast gas monitoring? Let’s use saturation arithmetic!</a></h1>
<h2 id="introduction-we-got-gas"><a class="toclink" href="#introduction-we-got-gas">Introduction: we got gas</a></h2>
<p>In Tezos, as with most smart contract platforms, on-chain operations cost <strong>gas</strong> — a theoretical resource intended to <a href="https://research-development.nomadic-labs.com/smarter-contracts-thanks-to-delphi-part-12.html">reflect (and so limit) the on-chain computational cost of running a smart contract</a>.</p>
<p>The <strong>gas model</strong> allocates gas costs to atomic computation steps.
When a computation starts it receives some finite allocation of gas, from which the gas cost of each of its atomic computations is deducted as the computation runs, step by step.
If the gas allocation is exhausted, the computation is deemed to have become too expensive and is aborted.<sup id="fnref:internship"><a class="footnote-ref" href="#fn:internship">1</a></sup></p>
<p>Three design considerations for a gas system are:</p>
<ol>
<li>
<p><em>The gas model should be accurate.</em></p>
<p>We don’t want easy computations to get terminated unnecessarily for lack of gas; and conversely, we don’t want the gas model to allow an attacker to run expensive computations whose cost is undetected by the gas model, thus opening a potential DoS attack surface.</p>
<p>Getting this right is an art: the recent Tezos Delphi protocol upgrade, for example, <a href="https://research-development.nomadic-labs.com/smarter-contracts-thanks-to-delphi-part-12.html">finessed gas costs significantly</a>.</p>
</li>
<li>
<p><em>The gas model should be computed (reasonably) correctly.</em></p>
<p>That is, the arithmetic of gas computations should be correctly performed. This sounds basic, but as we shall see, there is subtlety to this, because: </p>
</li>
<li>
<p><em>The gas model should be computed (fairly) cheaply.</em></p>
<p>Again, this sounds basic — but computing gas costs is a computation, just like anything else; and gas computations are ubiquitous on Tezos, so there may be a trade-off between efficiency and accuracy.</p>
</li>
</ol>
<p>Let’s talk about how we optimized the cost of gas monitoring while preserving its correctness, thanks to <strong>saturation arithmetic</strong>.
This optimization is a part of the <a href="http://doc.tzalpha.net/protocols/009_florence.html">Florence protocol proposal</a>, currently under consideration by the <a href="https://arxiv.org/pdf/1909.08458.pdf">Tezos on-chain voting procedure</a>.</p>
<h2 id="how-does-gas-monitoring-work"><a class="toclink" href="#how-does-gas-monitoring-work">How does gas monitoring work?</a></h2>
<p>Consider a node — call it <a href="https://en.wikipedia.org/wiki/Maschinenmensch">Maria</a> — on the Tezos network, about to perform a computation <span class="math">\(C\)</span> on some value <span class="math">\(V\)</span>.
Before computing <span class="math">\(C(V)\)</span>, Maria first uses a <strong>gas cost model</strong> <span class="math">\(M\)</span> to compute a gas quantity <span class="math">\(G_M(C,V)\)</span> which — if we designed our model correctly — fairly realistically anticipates the real-world cost of actually computing <span class="math">\(C(V)\)</span>.
Note that:</p>
<ul>
<li><span class="math">\(G_M\)</span> is determined by the cost model <span class="math">\(M\)</span>.</li>
<li><span class="math">\(G_M(C,V)\)</span> depends on the computation <span class="math">\(C\)</span>, and also on the value (or values) passed to it. This means that <span class="math">\(G_M(C,V)\)</span> must be computed at runtime, when <span class="math">\(V\)</span> is known, and cannot be statically determined at compile-time.<sup id="fnref:depending"><a class="footnote-ref" href="#fn:depending">2</a></sup></li>
</ul>
<p><span class="math">\(G_M(C,V)\)</span> has dimension <code>mgas</code> (<em>milligas</em>) units, and <span class="math">\(G_M(C,V)\)</span> <code>mgas</code> is subtracted from the <strong>remaining gas counter</strong>.</p>
<ul>
<li>If the remaining gas counter falls below 0, then Maria declines to compute <span class="math">\(C(V)\)</span>, due to it exhausting its available gas.</li>
<li>If the counter remains nonnegative, then Maria proceeds to compute <span class="math">\(C(V)\)</span>.</li>
</ul>
<p>Both the gas model and the check for exhaustion perform arithmetic operations over gas.
The usual arithmetic operations available in most CPUs implement <strong>modular arithmetic</strong> over 64-bit integers,
in which only the numbers from <span class="math">\(-2^{63}\)</span> to <span class="math">\(2^{63}-1\)</span> inclusive can be represented.</p>
<p>The OCaml <code>int</code> memory representation <a href="https://blog.janestreet.com/what-is-gained-and-lost-with-63-bit-integers/">reserves one bit for a compiler tag</a>.
Thus the effective length of an OCaml integer datum is 63 bits, and the native 64-bit machine instructions yield effective bounds of <span class="math">\(-2^{62}\)</span> to <span class="math">\(2^{62}-1\)</span> for a programmer using a signed integer in the OCaml compiler on a 64-bit system (or <span class="math">\(0\)</span> to <span class="math">\(2^{63}-1\)</span> for an <em>unsigned</em> integer).</p>
<p>If the result of an operation (e.g. add or multiply) exceeds these bounds, then we say the operation <a href="https://en.wikipedia.org/wiki/Integer_overflow"><strong>overflows</strong></a>.
The special case of an overflow <em>downwards</em> may also be called an <strong>underflow</strong>, and we will use this terminology in this post.</p>
<p>For example:</p>
<ul>
<li><span class="math">\((2^{62}-1) + 1\)</span> yields <span class="math">\(-2^{62}\)</span> (we overflow).</li>
<li><span class="math">\((-2^{62}) - 1\)</span> yields <span class="math">\(2^{62}-1\)</span> (we underflow).</li>
</ul>
<h2 id="overflow-is-dangerous"><a class="toclink" href="#overflow-is-dangerous">Overflow is dangerous!</a></h2>
<p>The gas model is public knowledge so an attacker knowing that <span class="math">\((2^{62}-1)+1 = -2^{62}\)</span> need only request an operation whose gas cost calculation triggers this addition, to make a <em>gas profit</em> of <span class="math">\(2^{62}\)</span> via overflow, thus literally counjuring gas out of nowhere and taking over the system.<sup id="fnref:flight-simulator"><a class="footnote-ref" href="#fn:flight-simulator">3</a></sup></p>
<h2 id="how-is-it-currently-implemented"><a class="toclink" href="#how-is-it-currently-implemented">How is it currently implemented?</a></h2>
<p>We currently use the arbitrary-precision integer <a href="https://github.com/ocaml/Zarith">ZArith OCaml library</a>, by Antoine Miné, Xavier Leroy, Pascal Cuoq, and Christophe Troestler.
<a href="https://tezos.gitlab.io/protocols/008_edo.html">Edo</a> performs gas computations using ZArith, and the <a href="https://gitlab.com/tezos/tezos/-/blob/master/src/proto_alpha/lib_protocol/script_interpreter.ml">Michelson interpreter</a> uses ZArith for its arithmetic on natural numbers, integers, and timestamps.</p>
<p>ZArith uses a dynamically-sized representation for numbers:
<em>any</em> number can be represented, provided it fits in the computer memory.
This is <em>safe</em>, but <em>slow</em>.
It protects Tezos from overflows by making them impossible, but:</p>
<ol>
<li>The representation is complex.
A mere machine word will not suffice, so arithmetic operations are costly simply because they process a more complex datastructure.</li>
<li>ZArith is implemented as a hybrid OCaml/C library. Calling a C function from OCaml is expensive. ZArith tries to avoid C functions when dealing with small integers, but it cannot fully escape this cost.
Thus even an addition over a 63-bit integer represented in ZArith is about five times slower than a typical addition over <code>int</code> type in OCaml.</li>
<li>Inevitably and by design, very large numbers are there to be computed on, and the computational cost scales with their size.</li>
</ol>
<h2 id="262-1-mgas-ought-to-be-enough-for-anybody"><a class="toclink" href="#262-1-mgas-ought-to-be-enough-for-anybody"><a href="https://en.wikiquote.org/wiki/Bill_Gates#Misattributed"><span class="math">\(2^{62} - 1\)</span> <code>mgas</code> ought to be enough for anybody</a></a></h2>
<p>From the <a href="https://tezos.gitlab.io/protocols/007_delphi.html">Delphi protocol</a> onwards — thus: including the <a href="https://tezos.gitlab.io/protocols/008_edo.html">current Edo protocol</a>, and also in the <a href="https://tezos.gitlab.io/protocols/009_florence.html">Florence protocol proposal</a> — an operation cannot consume more than <span class="math">\(1,040,000,000 \approx 2^{30}\)</span> <code>mgas</code>.
So representing <code>mgas</code> with a native OCaml 63 bit integer seems reasonable;
on a 64-bit architecture the integer limit of <span class="math">\(2^{62} - 1 = 4,611,686,018,427,387,903\)</span> <code>mgas</code> would be hit in future evolutions of the protocol only if we multiply the gas limit by several billion.<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">4</a></sup></p>
<p>Once the cost model reaches <span class="math">\(1,040,000,000\)</span> <code>mgas</code> the operation will be cancelled, and that’s all we need to know.
So integer <em>precision</em> is not an issue and we just need to consider the potential for over- or underflow.</p>
<p>Saturation arithmetic equips bounded integers with operations that do not overflow/underflow, but instead might <strong>saturate</strong> — they default to the biggest/smallest value that fits within the bounds (as we discuss next).</p>
<p>Because <span class="math">\(2^{30}<2^{62}\)</span>, our final results cannot saturate, and while there might exist intermediate computations that could saturate, we commit to writing code that will detect this and correct as necessary.</p>
<h2 id="how-to-implement-saturation-arithmetic"><a class="toclink" href="#how-to-implement-saturation-arithmetic">How to implement saturation arithmetic?</a></h2>
<p>Saturation arithmetic is a well-known technique to avoid underflow and overflow while computing with bounded integers.
In case of an overflow / underflow, the result is replaced by the maximum / minimum value.
We only need nonnegative integers for gas — remember if we drop below 0 we stop computing — so the minimum is <code>0</code> and the maximal is <code>max_int</code>, which in <a href="https://caml.inria.fr/pub/docs/manual-ocaml/libref/Int.html">OCaml running on a 64-bit system</a> is <span class="math">\(2^{62} - 1\)</span>. Thus:</p>
<ul>
<li><span class="math">\((2^{62}-1)+1\)</span> yields <span class="math">\(2^{62}-1\)</span>.</li>
<li><span class="math">\(0-1\)</span> yields <span class="math">\(0\)</span>.</li>
</ul>
<p>Saturation arithmetic operations are available in machine instruction sets like <span class="caps">MMX</span> or <span class="caps">AVX2</span>.
However, we prefer to implement them in software on top of standard modular arithmetic operations, for portability (the efficiency hit is modest).</p>
<p>The challenge is therefore to efficiently detect when an overflow or underflow has occurred:</p>
<ul>
<li>For addition: for <span class="math">\(0\leq x,y<2^{62}\)</span>, we have that <span class="math">\(x+y<0\)</span> if and only if the computation of <span class="math">\(x+y\)</span> overflows (where here <span class="math">\(+\)</span> denotes the modular arithmetic operation). Thus to check whether an addition has overflowed, we just check that the result is nonnegative. </li>
<li>For subtraction: for <span class="math">\(0\leq x,y<2^{62}\)</span>, we have that <span class="math">\(x-y<0\)</span> if and only if the computation <span class="math">\(x-y\)</span> underflows. Thus to check whether a subtraction has underflowed, we just check that the result is nonnegative.</li>
<li>Addition cannot underflow, and subtraction cannot overflow. Division can neither overflow nor underflow. </li>
<li>The interesting case is multiplication, as we now discuss.</li>
</ul>
<p>We use <a href="https://gitlab.com/tezos/tezos/-/blob/master/src/proto_009_PsFLoren/lib_protocol/saturation_repr.ml">a bespoke OCaml module <code>saturation_repr.ml</code></a> (see lines <a href="https://gitlab.com/tezos/tezos/-/blob/master/src/proto_009_PsFLoren/lib_protocol/saturation_repr.ml#L73">73</a> and
<a href="https://gitlab.com/tezos/tezos/-/blob/master/src/proto_009_PsFLoren/lib_protocol/saturation_repr.ml#L98">98</a>):</p>
<div class="highlight"><pre><span></span><code> <span class="mi">1</span> <span class="k">let</span> <span class="n">saturated</span> <span class="o">=</span> <span class="n">max_int</span>
<span class="mi">2</span>
<span class="mi">3</span> <span class="k">let</span> <span class="n">small_enough</span> <span class="n">z</span> <span class="o">=</span>
<span class="mi">4</span> <span class="n">z</span> <span class="ow">land</span> <span class="mh">0x7fffffff80000000</span> <span class="o">=</span> <span class="mi">0</span>
<span class="mi">5</span>
<span class="mi">6</span> <span class="k">let</span> <span class="n">mul</span> <span class="n">x</span> <span class="n">y</span> <span class="o">=</span>
<span class="mi">7</span> <span class="c">(* assert (x >= 0 && y >= 0); *)</span>
<span class="mi">8</span> <span class="k">match</span> <span class="n">x</span> <span class="k">with</span>
<span class="mi">9</span> <span class="o">|</span> <span class="mi">0</span> <span class="o">-></span>
<span class="mi">10</span> <span class="mi">0</span>
<span class="mi">11</span> <span class="o">|</span> <span class="n">x</span> <span class="o">-></span>
<span class="mi">12</span> <span class="k">if</span> <span class="n">small_enough</span> <span class="n">x</span> <span class="o">&&</span> <span class="n">small_enough</span> <span class="n">y</span> <span class="k">then</span> <span class="n">x</span> <span class="o">*</span> <span class="n">y</span>
<span class="mi">13</span> <span class="k">else</span> <span class="k">if</span> <span class="nn">Compare</span><span class="p">.</span><span class="nn">Int</span><span class="p">.</span><span class="o">(</span><span class="n">y</span> <span class="o">></span> <span class="n">saturated</span> <span class="o">/</span> <span class="n">x</span><span class="o">)</span> <span class="k">then</span> <span class="n">saturated</span>
<span class="mi">14</span> <span class="k">else</span> <span class="n">x</span> <span class="o">*</span> <span class="n">y</span>
</code></pre></div>
<p>A multiplication between two nonnegative integers overflows when <code>y > saturated / x</code>.
We see this check on line <code>13</code> above, but it requires a slow division so we try to avoid it on line <code>12</code> with a fast <a href="https://gitlab.com/tezos/tezos/-/blob/master/src/proto_009_PsFLoren/lib_protocol/saturation_repr.ml#L73"><code>small_enough</code></a> bitmask test that is sound, but not complete — an integer that passes <code>small_enough</code> is certainly small enough, but some integers are close enough to the bound that they may need re-checked the slow way.
In practice, most executions run through the fast path.</p>
<p>An additional implementation trick: we use a <a href="https://www.cs.ox.ac.uk/ralf.hinze/publications/With.pdf">phantom type</a>
to statically track the integers that are known to be safe for multiplication.
The clients of the saturation arithmetic module are then offered <a href="https://gitlab.com/tezos/tezos/-/blob/master/src/proto_009_PsFLoren/lib_protocol/saturation_repr.ml">several multiplications</a>
depending on the static knowledge they have about their arguments:</p>
<div class="highlight"><pre><span></span><code><span class="c">(** [mul x y] behaves like multiplication between native integers as</span>
<span class="c"> long as its result stays below [saturated]. Otherwise, [mul] returns</span>
<span class="c"> [saturated]. *)</span>
<span class="k">val</span> <span class="n">mul</span> <span class="o">:</span> <span class="o">_</span> <span class="n">t</span> <span class="o">-></span> <span class="o">_</span> <span class="n">t</span> <span class="o">-></span> <span class="n">may_saturate</span> <span class="n">t</span>
<span class="c">(** [mul_safe x] returns a [mul_safe t] only if [x] does not trigger</span>
<span class="c"> overflows when multiplied with another [mul_safe t]. *)</span>
<span class="k">val</span> <span class="n">mul_safe</span> <span class="o">:</span> <span class="o">_</span> <span class="n">t</span> <span class="o">-></span> <span class="n">mul_safe</span> <span class="n">t</span> <span class="n">option</span>
<span class="c">(** [mul_fast x y] exploits the fact that [x] and [y] are known not to</span>
<span class="c"> provoke overflows during multiplication to perform a mere</span>
<span class="c"> multiplication. *)</span>
<span class="k">val</span> <span class="n">mul_fast</span> <span class="o">:</span> <span class="n">mul_safe</span> <span class="n">t</span> <span class="o">-></span> <span class="n">mul_safe</span> <span class="n">t</span> <span class="o">-></span> <span class="n">may_saturate</span> <span class="n">t</span>
<span class="c">(** [scale_fast x y] exploits the fact that [x] is known not to</span>
<span class="c"> provoke overflows during multiplication to perform a</span>
<span class="c"> multiplication faster than [mul]. *)</span>
<span class="k">val</span> <span class="n">scale_fast</span> <span class="o">:</span> <span class="n">mul_safe</span> <span class="n">t</span> <span class="o">-></span> <span class="o">_</span> <span class="n">t</span> <span class="o">-></span> <span class="n">may_saturate</span> <span class="n">t</span>
</code></pre></div>
<h2 id="what-are-the-gains"><a class="toclink" href="#what-are-the-gains">What are the gains?</a></h2>
<p>Here are the running times of a <a href="https://gitlab.com/-/snippets/2095384">micro-benchmark</a>
which does 10,000 sequences of 10 operations over random pairs of integers:<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">5</a></sup></p>
<div class="highlight"><pre><span></span><code>┌────────────────┬────────────┐
│ Name │ Time/Run │
├────────────────┼────────────┤
│ Zarith add │ 2,993.12us │
│ Standard add │ 316.32us │
│ Saturation add │ 411.82us │
└────────────────┴────────────┘
┌────────────────┬────────────┐
│ Name │ Time/Run │
├────────────────┼────────────┤
│ Zarith sub │ 2,340.94us │
│ Standard sub │ 322.81us │
│ Saturation sub │ 421.24us │
└────────────────┴────────────┘
┌────────────────┬────────────┐
│ Name │ Time/Run │
├────────────────┼────────────┤
│ Zarith mul │ 3,285.63us │
│ Standard mul │ 321.70us │
│ Saturation mul │ 479.08us │
└────────────────┴────────────┘
┌────────────────┬────────────┐
│ Name │ Time/Run │
├────────────────┼────────────┤
│ Zarith div │ 481.98us │
│ Standard div │ 432.28us │
│ Saturation div │ 436.89us │
└────────────────┴────────────┘
</code></pre></div>
<p><a name="speedup"></a>
For addition, subtraction, and multiplication, </p>
<ul>
<li>the ratio between modulo and ZArith arithmetic operations is roughly <span class="math">\(10\)</span>, while </li>
<li>the ratio between modulo and saturation arithmetic operations is roughly <span class="math">\(1.25\)</span>. </li>
</ul>
<p>Unsurprisingly, there is no significant performance difference for division.</p>
<p>We see above that </p>
<ul>
<li>saturation arithmetic is almost as fast as standard modulo arithmetic, and</li>
<li>provided you steer clear of division, it is ten times faster than ZArith.<sup id="fnref:unfair"><a class="footnote-ref" href="#fn:unfair">6</a></sup></li>
</ul>
<p>In the Michelson interpreter (<a href="https://tezos.gitlab.io/008/michelson.html">documentation</a>; <a href="https://hackernoon.com/how-to-use-the-michelson-programming-language-to-write-smart-contracts-on-tezoss-y78f30e4">nice overview</a>) every execution cycle starts with a gas monitoring operation, and using saturation arithmetic makes this operation quicker:
the gas counter can be represented in a machine register (which was not the case in Edo using ZArith); the OCaml compiler can inline the saturation arithmetic operations; and the cost model execution is faster because it uses a more efficient arithmetic.
These optimizations taken together reduced the real computational cost of the Michelson execution cycle, and correspondingly allowed us to reduce by around 35% its gas cost in the Florence gas model.</p>
<p>When the node executes the following three well-known contracts, the gas consumed by the execution cycle<sup id="fnref:gascost"><a class="footnote-ref" href="#fn:gascost">7</a></sup> is measured as follows (units are milligas; <code>mgas</code>):</p>
<div class="highlight"><pre><span></span><code>┌──────────┬────────┬──────────┐
│ Contract │ Edo │ Florence │
├──────────┼────────┼──────────┤
│ Dexter │ 44,867 │ 28,813 │
│ FA1.2 │ 9,718 │ 6,238 │
│ Manager │ 3,764 │ 2,301 │
└──────────┴────────┴──────────┘
</code></pre></div>
<p>We used a bespoke version of Tezos to isolate the relevant signal in the measurement above, which we are happy to make <a href="https://gitlab.com/nomadic-labs/tezos/-/tree/yrg@instrument-gas-consumption">freely available for your convenience</a>.
Code pointers for the contracts above are:</p>
<ul>
<li><a href="https://gitlab.com/-/snippets/2099510">The Dexter (v1) contract (script)</a>.</li>
<li><a href="https://gitlab.com/-/snippets/2099511">The <span class="caps">FA1</span>.2 contract (script)</a>.</li>
<li>For the Manager contract, see <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/blob/master/src/contracts/manager.tz">this Manager script from the Mi-Cho-Coq repo</a>.</li>
</ul>
<h2 id="what-are-the-risks"><a class="toclink" href="#what-are-the-risks">What are the risks?</a></h2>
<p>The system is still protected from an attacker armed with a hostile operation that requires a very large amount of gas:
the cost model will saturate; the gas counter will be set to 0; and the hostile operation will be cancelled.
So saturation arithmetic protects the node from this type of attack just as well as ZArith.</p>
<p>However, a more subtle issue is that we lose some standard arithmetic identities, because saturation arithmetic operations treats saturated values just like any other value
— <code>max_int</code> and <code>0</code> serve double duty as themselves, and as overflow values, and this distinction is not recorded.
For example: </p>
<ul>
<li><span class="math">\((x - y) + y = x\)</span> is invalid in general since <span class="math">\(x - y\)</span> can saturate to <span class="math">\(0\)</span>. For example, <code>(0-1)+1 = 1</code>.</li>
<li><span class="math">\((x + y) - y = x\)</span> is invalid in general because <span class="math">\(x + y\)</span> can saturate to <code>max_int</code>. For example, <code>(1 + max_int) - max_int = 0</code>.</li>
<li>We lose the ability to rebracket in general. For example, <code>(0-1)+1=1</code> whereas <code>0-(1-1)=0</code>. </li>
</ul>
<p>This makes saturation arithmetic potentially counterintuitive, which could in principle lead to programmer error.
For instance, a programmer needs to be aware of situations where an intermediate computation might saturate (even if the final result is expected to be in bounds).
However, the risk of such error seems fairly low, for one general reason and one specific reason: </p>
<ul>
<li><em>The general reason</em> is that OCaml is a strongly-typed programming language and integers equipped with saturation arithmetic are represented in OCaml by an abstract type, such that the programmer is reminded (by the type) that these are saturation arithmetic integers, and only saturation arithmetic operations can be used to manipulate its values. </li>
<li><em>The specific reason</em> is that the cost model implementation <em>should</em> only need increasing functions, so the pitfalls described above should not arise
as long as the functions are written to correctly propagate saturation.
Aside from this, the rest of the gas monitoring subsystem is just trying to efficiently decrement a nonsaturated gas counter; if it saturates to 0, the operation is cancelled.</li>
</ul>
<p>So in practice, saturation arithmetic is a good fit for our needs, and adds efficiency. </p>
<h2 id="conclusions-and-future-work"><a class="toclink" href="#conclusions-and-future-work">Conclusions and future work</a></h2>
<p>So in conclusion: the 63-bit native integers of OCaml running on a 64-bit system are precise enough for gas monitoring using saturation arithmetic operations.
Moving from a general-purpose arbitrary-precision C library in Edo, to a <a href="https://gitlab.com/tezos/tezos/-/blob/master/src/proto_009_PsFLoren/lib_protocol/saturation_repr.ml">bespoke OCaml module</a> implementing saturation arithmetic operations in Florence, allowed us to</p>
<ul>
<li>speed up gas arithmetic by a factor of ten,</li>
<li>increase overall performance<sup id="fnref:3"><a class="footnote-ref" href="#fn:3">8</a></sup> of an execution cycle of the Michelson interpreter in Florence by 35%, </li>
<li>and thus globally improve the efficiency of the gas monitoring subsystem.</li>
</ul>
<p>For future work: </p>
<ul>
<li>
<p>We believe that saturation arithmetic could be further optimized by making it <a href="https://web.archive.org/web/20190213215419/https://locklessinc.com/articles/sat_arithmetic/">branch-free</a>.</p>
</li>
<li>
<p>Some automatic verification would be useful to further eliminate risk, by checking that the cost model implementation only uses increasing functions (and raise an alert if non-increasing functions are introduced).</p>
</li>
</ul>
<p>You can also read more about Nomadic Labs’ work on gas costing in a <a href="https://research-development.nomadic-labs.com/smarter-contracts-thanks-to-delphi-part-12.html">previous blog post on Delphi</a>.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:internship">
<p>So gas is a proxy for computational complexity, like money is a proxy for value. The gas model is like a pricelist, and the gas allocation is a budget. Computations that exceed their budget, get shut down. Nomadic Labs is advertising <a href="https://nomadic-labs.com/pages/adversarial-program-generator-for-gas-model-validation.html">an open internship on gas model validation</a>. If interested, please e-mail <a href="mailto:careers@nomadic-labs.com">careers@nomadic-labs.com</a>. <a class="footnote-backref" href="#fnref:internship" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:depending">
<p><a href="https://gitlab.com/morley-framework/morley/-/blob/e2d3cf9197804a2cfa92715e1a4d640fe5b87af7/docs/gasConsumption.md#types-of-costs">Gas costs depending on values</a> is common for Michelson because we have built-in data structures and arbitrary-precision numbers. Contrast with e.g. the Ethereum Virtual Machine’s gas model (<span class="caps">EVM</span>), in which <a href="https://docs.google.com/spreadsheets/d/1m89CVujrQe5LAFJ8-YAUCcNK950dUzMQPMJBxRtGCqs/edit#gid=0">almost all instructions have a fixed gas cost</a>. <a class="footnote-backref" href="#fnref:depending" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:flight-simulator">
<p>Early flight simulators were vulnerable to integer underflow: if you dived towards the ground and picked up enough downward speed, you could underflow the speed counter and find yourself travelling upwards at incredible velocity. It was possible using this trick to “hop” indefinitely with an empty fuel tank. Unfortunately <a href="https://en.wikipedia.org/wiki/Cluster_(spacecraft)">this issue was not restricted to games</a>. <a class="footnote-backref" href="#fnref:flight-simulator" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:1">
<p>… but not on a 32-bit architecture, since <span class="math">\(log_2 (1040000000) = 29.954\)</span>.
The Tezos node only runs on 64-bit architectures so that is not a problem. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>We compose several operations to reach situations
where large numbers appear in ZArith. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:unfair">
<p>In fairness to ZArith, it is good at what it does. But we only need to count to <span class="math">\(2^{30}\)</span>, so we just don’t need all the power that it offers. <a class="footnote-backref" href="#fnref:unfair" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:gascost">
<p>In practice, the cost of the Michelson interpreter cycle is only part of the full picture,
and other costs may intervene (as for any large system), e.g. deserialisation costs.
We are of course actively optimising these too
(for instance, we recently optimised deserialisation costs by a factor of around 10,
see this <a href="https://gitlab.com/nomadic-labs/data-encoding/-/merge_requests/23"><span class="caps">MR</span></a>). <a class="footnote-backref" href="#fnref:gascost" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>The speedup per arithmetic operation from ZArith to saturation arithmetic <a href="#speedup">is approximately 10, as observed above</a>. The execution cycle includes other operations — typically dispatching over instructions and moving to the next instruction — so the overall reduction is 35%. <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Tezos calling convention migrating from Breadth-First to Depth-First Order (BFS to DFS)2021-03-08T18:00:00+01:002021-03-08T18:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2021-03-08:/tezos-calling-convention-migrating-from-breadth-first-to-depth-first-order-bfs-to-dfs.html<p><em><strong>Summary: If the Florence proposal is adopted, we recommend you do not deploy new Michelson contracts that are dependent on the <span class="caps">BFS</span> calling convention. We do not expect this to be a problem in practice. However, those planning on deploying contracts in the near term should check that their contract’s correctness is unaffected by the change in calling convention.</strong></em></p>
<p>The current calling convention for intercontract calls in Tezos is that they are added to a “first-in, first-out” queue, also called a “<span class="caps">BFS</span>” (or “breadth-first”) approach. </p>
<p>The proposed Florence protocol update includes a switch to a “first-in, last-out” convention, also called a “<span class="caps">DFS</span>” (“depth-first”) approach.</p>
<p>Early in the history of Tezos, the decision was made to use a <span class="caps">BFS</span> calling convention for intercontract calls. This was motivated by theoretical work that appeared to show that <span class="caps">BFS</span> would be superior to <span class="caps">DFS</span> for this purpose. </p>
<p>However, experience indicates the opposite: </p>
<ul>
<li>the <span class="caps">BFS</span> calling convention can confuse some developers and cause errors, and </li>
<li>it complicates porting contracts from other chains, where <span class="caps">DFS</span> calling conventions dominate.</li>
</ul>
<p><a href="https://gitlab.com/tzip/tzip/-/merge_requests/111">A number of mechanisms</a> were considered for the Florence proposal to add a <span class="caps">DFS</span> calling convention in addition to <span class="caps">BFS</span>. However, when mechanisms for backwards compatibility with legacy contracts were considered, subtle bugs were frequently found that would impact the correctness of existing contracts, and which could also render future contracts unsafe.</p>
<p>In practice reasoning about a mixed calling convention is hard, and accordingly the Florence proposal contains a straightforward change to switch the Tezos intercontract calling convention from <span class="caps">BFS</span> to <span class="caps">DFS</span>.</p>
<p>Replays of on-chain history indicate that this migration does not break the ordinary functionality of any existing live contracts. Because the <span class="caps">BFS</span> convention typically provides few useful guarantees, it appears that current contracts deployed on the chain are insensitive to calling order (we speculate that authors just found reasoning about <span class="caps">BFS</span> calling order too difficult, so avoided depending on it).</p>
<p>However, new contracts with a calling order dependency might get added to the chain between the injection of Florence and its adoption. Therefore, we recommend you do not add new contracts that <em><strong>depend</strong></em> on <span class="caps">BFS</span> calling conventions to the chain — if Florence is adopted, such contracts would break.</p>
<p>Note that, <strong>unless you explicitly built a contract to depend on the <span class="caps">BFS</span> calling convention, it probably doesn’t</strong>.</p>
<p>We will continue to monitor the chain for new contracts that might depend on calling convention ordering, and attempt to contact their authors.</p>
<p>Although this upgrade path is imperfect, it seems the best available option for improving both the smart contract developer experience and the future safety of contracts deployed on the Tezos network. The alternatives would have increased complexity and created unacceptable scope for error.</p>
<p>The <span class="caps">BFS</span> calling convention for smart-contract interactions was an unfortunate design flaw. However, Tezos can self-amend and is always evolving, so we in the Tezos community can solve this issue by adopting the Florence amendment through an on-chain vote.</p>Baking Accounts proposal contains unexpected breaking changes2021-03-08T17:00:00+01:002021-03-08T17:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2021-03-08:/baking-accounts-proposal-contains-unexpected-breaking-changes.html<h2 id="summary"><a class="toclink" href="#summary">Summary</a></h2>
<p>Ongoing testing and review of baking accounts has uncovered some important and previously undocumented breaking changes (see <a href="https://gitlab.com/tzip/tzip/-/merge_requests/133/diffs#9eb8e126493e85d8c9f2b71eb47cee980df5f5c2_0_524">the section on breaking changes in the <span class="caps">TZIP</span> for Baking Accounts</a>) in the baking account proposal. </p>
<p>These issues are significant, and affect the functionality of both existing and future smart contracts; they are detailed below. Bakers should please these carefully when casting their vote.</p>
<p><em>We believe Baking Accounts should be postponed until a thorough audit of functionality is complete, or an alternative implementation produced. <a href="https://www.tezosagora.org/proposal/11">The version of Florence without baking accounts</a> is a safer choice.</em></p>
<h2 id="source-and-sender-changes"><a class="toclink" href="#source-and-sender-changes"><span class="caps">SOURCE</span> and <span class="caps">SENDER</span> changes</a></h2>
<h3 id="consensus-keys-can-not-be-sender-or-source"><a class="toclink" href="#consensus-keys-can-not-be-sender-or-source">Consensus keys can not be <span class="caps">SENDER</span> (or <span class="caps">SOURCE</span>)</a></h3>
<p>After the migration to Florence, a baker’s consensus key can never be <code>SENDER</code> (or <code>SOURCE</code>). This means they cannot authenticate themselves in the usual way to smart contracts (through <code>SENDER</code>, though some contracts incorrectly use <code>SOURCE</code>).</p>
<p>This includes all current delegate keys, including those of inactive delegates. These keys will become consensus keys upon the migration.</p>
<p>In particular, if any tokens (<span class="caps">FA1</span>.2, <span class="caps">FA2</span>, or similar) are sent to consensus keys, or already owned by delegates upon the protocol migration, these <strong>tokens will be locked</strong>, unless an allowance or an operator was set up before the migration.</p>
<p>Any <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/blob/master/src/contracts/manager.tz">manager.tz</a> contracts (like those created automatically from originated “accounts” during the Babylon migration) which are managed by delegates </p>
<ul>
<li>will become <strong>inoperable</strong>, and </li>
<li>their <strong>tez will be locked</strong>,</li>
</ul>
<p>unless emptied before the migration. Roughly 85,000 ꜩ are currently in this state.</p>
<p>The “bakers registry” contract would need to be updated and redeployed to account for this, since it expects a baker to use their key hash as <code>SENDER</code> at least once (to set a “reporter” address). A new version of this contract should use the baker hash instead.</p>
<p>There could be other examples. For instance: any contract with an “owner” or “admin” address set to a delegate key would become impossible to administrate upon the protocol migration.</p>
<p>Rotating the consensus key out by setting a different one does <em>not</em> solve this problem. Any consensus key is <em>permanently</em> unable to be <code>SENDER</code> or <code>SOURCE</code>, until a future protocol fixes the problem.</p>
<h3 id="source-might-not-be-an-implicit-account"><a class="toclink" href="#source-might-not-be-an-implicit-account"><span class="caps">SOURCE</span> might not be an implicit account</a></h3>
<p>The <code>SOURCE</code> instruction normally returns the address of the account responsible for invoking the call in the first place. Since the Babylon upgrade, the invariant that <code>SOURCE</code> is always an implicit account has held.</p>
<p>If a baker consensus key is used to initiate a transaction, the <code>SOURCE</code> instruction in Michelson will return the baking account address, and not the address of an implicit account.</p>
<p>Code that relies on <code>SOURCE</code> necessarily being an implicit account would be broken. We are not currently aware of any contract that relies on this behavior.</p>
<h3 id="source-sender-does-not-imply-a-top-level-call"><a class="toclink" href="#source-sender-does-not-imply-a-top-level-call"><span class="caps">SOURCE</span> = <span class="caps">SENDER</span> does not imply a top-level call</a></h3>
<p>The interaction with <code>SENDER</code> is potentially more problematic. In the case of the consensus key being used as part of a multisig call to trigger the baking contract to send out transaction, the instruction <code>SENDER</code> would also return the baking account address.</p>
<p>A common pattern is to use <code>SOURCE = SENDER</code> to ensure that a transaction is happening at the top-level and cannot be interleaved with other calls. With baking accounts, it would no longer be true that <code>SOURCE = SENDER</code> implies a top-level call. Indeed, using the consensus key to trigger a baking account to release a series of transactions creates a non-top-level call where <code>SOURCE = SENDER</code> nonetheless.</p>
<p>However, given the change to the <span class="caps">DFS</span> calling convention in Florence, it’s no longer possible to use a top-level call to interleave transactions, which seemingly mitigates the issue.</p>
<p>A design where <code>SOURCE</code> represents the implicit account associated with the consensus key and <code>SENDER</code> the baking account would be more consistent. It could be folded in later on, so long as no application comes to depends on the semantics detailed above.</p>
<h2 id="calls-to-implicit-accounts-can-fail"><a class="toclink" href="#calls-to-implicit-accounts-can-fail">Calls to implicit accounts can fail</a></h2>
<p>Calls to implicit accounts never fail when the burn limit is sufficiently high, the gas limit is sufficiently high, and the amount is positive.</p>
<p>In baking accounts, when a baker rotates to a new key, there is a period of time where the key is pending. During that period, transfers to that implicit account fail.</p>
<p>This can break contracts. For instance, the <a href="https://github.com/tzcolors/tzcolors-contract/blob/main/auction_house.py">auction contract</a> from <a href="https://www.tzcolors.io/">tzcolors</a> breaks as follows: A user could maliciously create an implicit account, bid on an item, delete the implicit account, and then register the account as a consensus key. If someone tries to outbid them, the transaction would involve a transfer back to the current winning bid, which would fail as it is the pending consensus key of a baking account. This would make it impossible to outbid the malicious bid.</p>
<p>While it might be possible for tzcolors to migrate their auction contract in the interim, the functionality could not be replicated with the baking account proposal. Instead, participants who are outbid would need to manually claim their bids back. Other versions of this contract are also deployed on the chain and would face the same issue.</p>
<h2 id="create_contract-fails-with-some-legacy-code"><a class="toclink" href="#create_contract-fails-with-some-legacy-code">CREATE_CONTRACT fails with some legacy code</a></h2>
<p>The baking account proposal changes the Michelson instruction <code>SET_DELEGATE</code> to take a <code>option baking_hash</code> instead of a <code>option key_hash</code>. Contracts which rely on a <code>key_hash</code> are marked as legacy, and can continue to use the old instruction, but new contracts can only use the new version of the <code>SET_DELEGATE</code> instruction.</p>
<p>This breaks contracts that work as contract factories, if they try to automatically deploy new contracts with the legacy <code>SET_DELEGATE</code> instruction. This includes <a href="https://kolibri.finance/">Kolibri</a> and <a href="https://docs.stakerdao.com/wrapped-tezos-wxtz-faq-and-docs">wXTZ</a>. Both contracts are upgradeable and could adapt to the change, but the activation of the baking account proposal could distrupt their operation.</p>
<h2 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h2>
<p>Due to these breaking changes, we believe that </p>
<ul>
<li>baking accounts are not currently appropriate for on-chain use, and</li>
<li>it is safest to accept <a href="https://www.tezosagora.org/proposal/11">the proposal without them</a>. </li>
</ul>
<p>A future version of Baking Accounts which does not break current contracts and preserves important invariants is possible, and should be developed to take its place.</p>
<p>Baking accounts is a feature whose design and implementation have proven a significant challenge, because: </p>
<ul>
<li>it requires extensive code updates; and </li>
<li>it changes the delegation system and thus how Tezos as a blockchain operates.</li>
</ul>
<p>This underscores that Tezos development needs to provide more room for specifying precisely and systematically how new features integrate with the existing codebase. The <a href="https://gitlab.com/tzip/tzip">Tezos Improvement Proposals (TZIPs) process</a> is a much-needed step in this direction, and we are acting to integrate this even more tightly into our development process. </p>
<p>By conditioning the implementation and integration of new features on the formulation of comprehensive, community-reviewed specifications, we increase our chances of catching issues like those presented in this article. From this perspective, the <a href="https://gitlab.com/tzip/tzip/-/merge_requests/133">Baking accounts <span class="caps">TZIP</span></a> was a missed opportunity, and we believe it is in our interest as a community to allocate more time and attention to this process.</p>Florence: Our Next Protocol Upgrade Proposal2021-03-04T15:30:00+01:002021-03-04T15:30:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2021-03-04:/florence-our-next-protocol-upgrade-proposal.html<p>Announcing Florence Proposal</p><p><span class="caps">UPDATE</span>: We believe that the baking accounts implementation is significantly flawed. See: <a href="https://research-development.nomadic-labs.com/baking-accounts-proposal-contains-unexpected-breaking-changes.html">Baking Accounts proposal contains unexpected breaking changes</a></p>
<p><em>This is a joint announcement from Nomadic Labs, Marigold, DaiLambda, and Tarides.</em></p>
<p>As we described <a href="https://forum.tezosagora.org/t/regular-scheduling-for-our-tezos-proposals/2331">in this post</a>, several development organizations in the Tezos ecosystem are now collaborating to submit protocol upgrade proposals every few months, which is the interval permitted by the Tezos on-chain governance process. When the <a href="https://forum.tezosagora.org/t/edo-the-latest-tezos-upgrade-is-live/2722">Edo upgrade</a> went live on February 13, we mentioned that a new protocol proposal, codenamed “Florence”, would soon be ready. We are pleased to announce that, as expected, “Florence” is now complete.</p>
<p>We are offering the community two versions of the Florence proposal to choose between, one with Baking Accounts (as described below) and one without; we’ll explain the rationale for this decision later in this blog post.</p>
<ul>
<li>The hash of the proposal with baking accounts is: <code>PsFLorBArSaXjuy9oP76Qv1v2FRYnUs7TFtteK5GkRBC24JvbdE</code></li>
<li>The hash of the proposal without baking accounts is: <code>PsFLorenaUUuikDWvMDr6fGBRG8kt3e3D3fHoXK1j1BFRxeSH4i</code></li>
</ul>
<p>Florence has a number of bug fixes and small improvements; <a href="http://doc.tzalpha.net/protocols/009_florence.html">we encourage you to look at the change log</a>. Below we will discuss some of the more interesting and important changes:</p>
<p><em>Increased Maximum Operation Size:</em> Previously, the maximum size of an operation was 16kB. In Florence, we propose to increase it to 32kB. Among other things, this has the effect of slightly more than doubling the maximum size of a smart contract, which should be of interest to some developers with particularly complicated applications.</p>
<p><em>Gas Optimizations:</em> We have again reduced gas consumption in smart contract execution by increasing the efficiency of gas computation inside the Michelson interpreter. This allows for smart contracts with more complicated functionality to operate economically on the chain. We will continue to work on further efficiency improvements in coming versions of the protocol.</p>
<p><em>Baking Accounts:</em> Previously, token holders delegated to a baker by specifying that baker’s public key hash. This meant that bakers could never change their public keys, which was exceptionally inconvenient. The new “Baking Accounts” feature alleviates this issue. In Florence, a new account type has been added to represent accounts managed by bakers. These accounts are Michelson smart contracts running a fixed multisig script. This feature lets bakers renew and split their consensus keys without moving to a new address and asking their delegators to follow. In addition to the usual internal operations, baking accounts can also emit baking operations such as proposing and voting for protocol amendments.</p>
<p>(The rights granted to the baking key (baking, endorsing, voting, and spending the funds) remain unchanged. However, the system also allows a baker to vote and access their funds using multisig authentication. Not using the baking key for such tasks reduces the risk of it being exposed, and the baking key can also be rotated in a worst case scenario.)</p>
<p>Although we strongly believe that Baking Accounts are an important new addition to the Tezos protocol, and although we have made considerable effort to make them backwards compatible with existing code, we recognize that client libraries, wallets, indexers, and other software will require some work to fully support Baking Accounts. We are thus providing the community with the opportunity to decide for itself during the Proposal Period whether or not to include this feature in the Florence update. </p>
<p><em>Depth First Execution Order:</em> Previously, intercontract calls were executed in a so-called “breadth first” ordering. This was believed to be the correct choice when the Tezos protocol was initially designed, but it has turned out to significantly complicate the lives of smart contract developers. If Florence is adopted, the calling convention will change to a “depth first” execution order. This will make it far easier to reason about intercontract calls.</p>
<p><em>No More Test Chain:</em> Previously, during the voting process, a test chain would be spun up during the “testing period” which took place between the exploration and promotion voting periods. The intent was that this test chain be used to assure that the new proposal worked correctly, but in practice, the test chain has never been used in this manner, and has caused significant operational problems to node operators. The new proposal eliminates the test chain activation; the testing period has been retained but is now named the “cooldown period”. Instead, we will continue to test the protocol using test chains that operate outside of the mainnet voting process.</p>
<p>This protocol amendment and the related updates to the Tezos shell were developed by programmers from Nomadic Labs, Metastate, DaiLambda, Marigold, Tarides, and an external contributor, <a href="https://gitlab.com/keefertaylor">Keefer Taylor</a>, to whom the proposals grant an invoice of ꜩ100 to thank him for his merge request that increased the maximum operation size. </p>
<p>Now that Florence is code complete, we encourage you to test your own Tezos related applications to check for compatibility problems. Docker images with both proposals are now available and two testnets, one with and one without Baking Accounts, will soon be available as well.</p>
<h3 id="supplementary-information"><a class="toclink" href="#supplementary-information">Supplementary Information:</a></h3>
<p><a href="https://gitlab.com/tezos/tezos/-/tree/master/src/proto_009_PsFLorBA/lib_protocol">Gitlab Repository with Baking Accounts</a></p>
<p><a href="https://gitlab.com/tezos/tezos/-/tree/master/src/proto_009_PsFLoren/lib_protocol">Gitlab Repository without Baking Accounts</a></p>
<p>A docker image containing both proposals and the necessary code to run them may be obtained by running:</p>
<div class="highlight"><pre><span></span><code>docker pull tezos/tezos:master
</code></pre></div>
<ul>
<li>
<p>The test network for Florence will be called
<code>Florencenet</code>. This will running the proposal with new baking accounts feature, whose protocol hash is: <code>PsFLorBArSaXjuy9oP76Qv1v2FRYnUs7TFtteK5GkRBC24JvbdE</code></p>
</li>
<li>
<p>The test network for Florence without baking accounts will be called <code>FlorenceNoBAnet</code>. The hash of that protocol proposal is: <code>PsFLorenaUUuikDWvMDr6fGBRG8kt3e3D3fHoXK1j1BFRxeSH4i</code></p>
</li>
</ul>A technical description of the Dexter flaw2021-02-25T15:00:00+01:002021-02-25T15:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2021-02-25:/a-technical-description-of-the-dexter-flaw.html<p>In this technical blog post, we detail the flaw found in the Dexter contract and the exploit used to “white-knight” the funds in those contracts.</p>
<h2 id="background"><a class="toclink" href="#background">Background</a></h2>
<p>The Dexter contract contains several entrypoints allowing users to perform various operations, such as adding and removing liquidity, or converting tokens to tez back and forth. The exact interface is given by the type of the contract’s parameter:</p>
<div class="highlight"><pre><span></span><code><span class="k">parameter</span><span class="w"> </span><span class="p">(</span><span class="kt">or</span>
<span class="w"> </span><span class="p">(</span><span class="kt">or</span>
<span class="w"> </span><span class="p">(</span><span class="kt">or</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="k k-Name">%approve</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">:spender</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">:allowance</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">:currentAllowance</span><span class="p">)))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="k k-Name">%addLiquidity</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">:owner</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">:minLqtMinted</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">:maxTokensDeposited</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">timestamp</span><span class="w"> </span><span class="k k-Name">:deadline</span><span class="p">))))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">or</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="k k-Name">%removeLiquidity</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">:owner</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">:to</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">:lqtBurned</span><span class="p">)))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">mutez</span><span class="w"> </span><span class="k k-Name">:minXtzWithdrawn</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">:minTokensWithdrawn</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">timestamp</span><span class="w"> </span><span class="k k-Name">:deadline</span><span class="p">))))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">or</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="k k-Name">%xtzToToken</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">:to</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">:minTokensBought</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">timestamp</span><span class="w"> </span><span class="k k-Name">:deadline</span><span class="p">)))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="k k-Name">%tokenToXtz</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">:owner</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">:to</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">:tokensSold</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">mutez</span><span class="w"> </span><span class="k k-Name">:minXtzBought</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">timestamp</span><span class="w"> </span><span class="k k-Name">:deadline</span><span class="p">)))))))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">or</span>
<span class="w"> </span><span class="p">(</span><span class="kt">or</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="k k-Name">%tokenToToken</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">:outputDexterContract</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">:minTokensBought</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">:owner</span><span class="p">)))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">:to</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">:tokensSold</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">timestamp</span><span class="w"> </span><span class="k k-Name">:deadline</span><span class="p">))))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">or</span><span class="w"> </span><span class="p">(</span><span class="kt">key_hash</span><span class="w"> </span><span class="k k-Name">%updateTokenPool</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">%updateTokenPoolInternal</span><span class="p">)))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">or</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="k k-Name">%setBaker</span><span class="w"> </span><span class="p">(</span><span class="kt">option</span><span class="w"> </span><span class="kt">key_hash</span><span class="p">)</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">or</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">%setManager</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">unit</span><span class="w"> </span><span class="k k-Name">%default</span><span class="p">)))));</span>
</code></pre></div>
<p>Of particular interest is the <code>tokenToXtz</code> entrypoint, which allows the swapping of tokens for tez. In Tezos, entrypoints are represented as different cases of a single sum type representing the expected parameter. The type for <code>tokenToXtz</code> is given as </p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="k k-Name">%tokenToXtz</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">:owner</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">address</span><span class="w"> </span><span class="k k-Name">:to</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">:tokensSold</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="p">(</span><span class="kt">mutez</span><span class="w"> </span><span class="k k-Name">:minXtzBought</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">timestamp</span><span class="w"> </span><span class="k k-Name">:deadline</span><span class="p">)))))))</span>
</code></pre></div>
<p>This means that the entrypoint expects a record with the following fields</p>
<ul>
<li><code>deadline</code> : the time until which the transaction requested is valid</li>
<li><code>minXtzBought</code> : the minimum amount of tez requested. If the tokens were to be swapped for fewer tez, the transaction be considered invalid, protecting the seller against excessive slippage</li>
<li><code>tokensSold</code> : the number of tokens the seller desires to sell</li>
<li><code>to</code> : the address where the tez proceeds are to be sent</li>
<li><code>owner</code> : the owner of the tokens being sold</li>
</ul>
<p>When the Dexter contract receives requests, it performs a series of computations to determine how many tez should be sold and, ultimately, emits two operations. One which sends tez to the “to” field, and the other, directed at the <span class="caps">FA1</span>.2 token contract, which request a transfer of tokens from the <code>owner</code> to the Dexter contract itself. If any of these transactions fail, the whole transaction fails.</p>
<p>An <span class="caps">FA1</span>.2 transfer is valid if and only if the contract calling the <span class="caps">FA1</span>.2 contract owns the tokens being transferred, or if it is present in the allowance list for the owner of the tokens. This is a pattern found in <span class="caps">ERC20</span> as well.</p>
<p>When Dexter users sell tokens to Dexter, they typically first make a temporary allowance in the <span class="caps">FA1</span>.2 contract to allow Dexter to access their funds.</p>
<h2 id="flaw"><a class="toclink" href="#flaw">Flaw</a></h2>
<p>The presence of an <code>owner</code> field in <code>tokenToXtz</code> raises two problems. First, it means a user is able to spend the token of anyone who has an open allowance to Dexter. While it is possible to recommend that allowances only be set temporarily and be reset to 0 after every interaction with the contract, this was not, in fact, done, and we found that several liquidity providers had small dangling allowances left after adding liquidity.</p>
<p>More problematic is the fact that the <code>owner</code> field could simply be set to the Dexter contract’s address itself. Since the caller of an <span class="caps">FA1</span>.2 contract is always allowed to spend its own funds, this means that the Dexter contract could be instructed to dispense tez by sending itself its own tokens. Self-transfers are normal, and even required in the <span class="caps">FA2</span> standard.</p>
<p>Dexter implements a constant formula market maker which attempts to maintain constant (before fees) the product of the quantity of the two assets it holds. By asking the Dexter contract to send itself its entire token balance, it’s possible to collect (before fees) roughly half of its tez balance.</p>
<p>The presence of an <code>owner</code> field in <code>tokenToXtz</code> was unnecessary, and its only safe value would be <code>SENDER</code>, which renders it useless as a parameter.</p>
<h2 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h2>
<p>It is important for authors of smart contracts to consider and test edge conditions of all sorts on all possible calls. This is especially true in situations where a contract can be told to transfer value to or from a third party. Creators of contracts should strongly consider avoiding such facilities entirely when they are not critical to the function of a contract. When providing such facilities, it is vital to consider whether edge cases might result in unexpected behavior.</p>Dexter Flaw Discovered; Funds are Safe2021-02-20T04:00:00+01:002021-02-20T04:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2021-02-20:/dexter-flaw-discovered-funds-are-safe.html<p><strong><span class="caps">TL</span>;<span class="caps">DR</span>:</strong> <em>A flaw was found in the camlCase’s Dexter contract. The funds have been removed from the contract and returned to their original holders.</em></p>
<p>A high level explanation follows; technical details of the Dexter flaw will be described in a separate post to come.</p>
<p>As many of you know, we have been working on a new Tezos upgrade proposal. This proposal, if accepted, will change the calling convention from breadth first ordering to depth first ordering. In the course of reviewing potentially affected contracts, we stumbled into an unrelated but serious flaw in Dexter permitting unauthorized withdrawal of funds.</p>
<p>We provided information on the vulnerability to camlCase, and, following responsible disclosure policies, we have waited to make a public statement until public discussion of the vulnerability would no longer harm the participants in Dexter.</p>
<p>Since then, a so-called “White Knight” operation has been conducted in which the funds were removed from the contract using the bug itself and then returned to their rightful owners. We are informed that this operation has now concluded, and all funds are safe.</p>
<p>Given the apparent substantial community interest in availability of a distributed exchange contract, a team led by Nomadic Labs has, in the interim, rewritten the contract to avoid this bug, and is in the process of proving that this particular class of bugs cannot re-occur.
We will be releasing both the code and specification for this contract shortly.</p>
<p>When we first published <a href="https://research-development.nomadic-labs.com/dexter-decentralized-exchange-for-tezos-formal-verification-work-by-nomadic-labs.html">our blog post announcing the formal verification of the Dexter contract</a>, we wrote some important things about what we had verified:</p>
<blockquote>
<p>As with any formal verification, there are limits to our development. Some are inherent to software verification, while others stem from limitations of Mi-Cho-Coq.</p>
</blockquote>
<p>We then went on to list several limitations, including this:</p>
<blockquote>
<p>Formal verification of the soundness of Dexter’s specification.</p>
<p>By soundness we mean conformity to some “common sense” economic properties; e.g. that an attacker can’t remove the funds of a liquidity provider without permission.</p>
<p>This is simple garbage in, garbage out: we can check whether an implementation satisfies its specification, but what if the specification’s wrong? For instance, a specification’s author might inadvertently permit an attack by which an attacker would remove funds that are not theirs, by simple human error or by not fully understanding the domain logic of the system.</p>
</blockquote>
<p>Unfortunately, our words have proven prophetic.</p>
<p>We again note that formal verification can only check if a contract meets a given specification, but not if the specification is incorrect.</p>
<p>As we have noted, we will soon be publishing the specification of the new contract. We invite interested members of the community to examine the new specification and provide feedback on whether our belief that the proper safety and correctness properties are now present is correct.</p>
<p>We look forward to the deployment and adoption of this new distributed exchange contract by interested parties once the community is satisfied that it is secure.</p>The Protocol: from High-level Command Line to Low-level Operations2021-02-16T18:00:00+01:002021-02-16T18:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2021-02-16:/the-protocol-from-high-level-command-line-to-low-level-operations.html<p>Understanding the source code of Tezos may take much time for fresh contributors. This gentle note gives a global presentation of the control flow of a simple transaction example, going from the high-level command line call to the low-level account operations.</p><p>This note explains and illustrates flow of control in Tezos using the example of carrying out a simple transaction. We will go from the high-level <a href="#command">command line call</a> to the <a href="#operations">low-level account operations</a>.
To guide you through the codebase, we give line numbers based on <a href="https://gitlab.com/tezos/tezos/-/releases/8.2.0">release 8.2 (commit 6102c808)</a>.</p>
<h1 id="where-do-i-start"><a class="toclink" href="#where-do-i-start">Where do I start?</a></h1>
<p><strong>Our scenario.</strong> Imagine that Alice and Bob have one <a href="https://tezos.gitlab.io/alpha/glossary.html#account">account</a> each, and Alice wants to send ꜩ10 from her account to Bob’s.
Alice can proceed by using a <a href="https://tezos.gitlab.io/alpha/glossary.html#transaction">transfer operation</a>.
This transaction will trigger some additional transactions, which will be handled automatically by the system:</p>
<ol>
<li>Alice must pay <em>baking fees</em> (tokens paid to the <a href="https://tezos.gitlab.io/alpha/glossary.html#baker">baker</a> who writes Alice’s main transfer onto the blockchain).</li>
<li>Alice may also have to pay for <em>key revelation</em> to reveal the public key of her account.<sup id="fnref:revelation"><a class="footnote-ref" href="#fn:revelation">1</a></sup></li>
</ol>
<p>Here’s a picture:</p>
<p><img alt="" src="images/proto-controlflow-transfer/tezos-transfer-tokens.svg"></p>
<p><a id="command"></a>
The easiest way to execute our example is to run a node locally by using the <a href="http://tezos.gitlab.io/user/sandbox.html">sandboxed mode</a>.
In the terminal, we just type:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-client<span class="w"> </span>transfer<span class="w"> </span><span class="m">10</span><span class="w"> </span>from<span class="w"> </span>alice<span class="w"> </span>to<span class="w"> </span>bob
</code></pre></div>
<p><a id="operations"></a></p>
<h1 id="inside-the-code"><a class="toclink" href="#inside-the-code">Inside the code</a></h1>
<blockquote>
<p>As far as the Tezos codebase is concerned, everything is a contract.
This includes user-owned accounts (called <a href="https://tezos.gitlab.io/alpha/glossary.html#implicit-account"><em>implicit</em> accounts</a> in the codebase) and smart contracts (called <a href="https://tezos.gitlab.io/alpha/glossary.html#smart-contract"><em>originated</em> accounts</a> in the codebase).
This is reflected in the variable and function names in the code cited below, so for consistency we may also call everything a “contract” — but note that in our example, the source and destination contracts are actually the user-owned (i.e., implicit) accounts of Alice and Bob.</p>
</blockquote>
<h2 id="step-1-dispatching-the-client-command"><a class="toclink" href="#step-1-dispatching-the-client-command">Step 1: Dispatching the client command</a></h2>
<p><img alt="" src="images/proto-controlflow-transfer/tezos-transfer-tokens-src-client.svg"></p>
<ul>
<li>
<p>The flow of control starts with the <strong><code>commands</code></strong> function, whose code is in <a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_client_commands/client_proto_context_commands.ml"><code>client_proto_context_commands.ml</code></a> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_client_commands/client_proto_context_commands.ml#L254">l.254</a>).
This determines the command-line syntax for the switches following an invocation of ‘tezos-client’.<sup id="fnref:commands"><a class="footnote-ref" href="#fn:commands">2</a></sup>
In our example, we have invoked the <code>transfer</code> sub-command (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_client_commands/client_proto_context_commands.ml#L893">l.893</a>).</p>
</li>
<li>
<p>Control now flows to <strong><code>transfer_command</code></strong> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_client_commands/client_proto_context_commands.ml#L110">l.110</a>), which distinguishes the type of the source contract (in the pattern-matched variable <code>source</code><a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_client_commands/client_proto_context_commands.ml#L136">l.136</a>) — in our case it is implicit and admits a public key hash.
Flow of control passes to another function, <code>transfer</code> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_client_commands/client_proto_context_commands.ml#L167">l.167</a>).</p>
</li>
<li>
<p><strong><code>transfer</code></strong> is in <code>client_proto_context.ml</code> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_client/client_proto_context.ml#L73">l.73</a>).
Its job is to build the transaction operation itself, using a function <code>build_transaction_operation</code>.
In our case, the operation is a <span class="caps">GADT</span> <code>Transaction</code> of type <code>manager_operation</code> (<a href="#manager_operation">discussed below</a>).
In particular, its field <code>amount</code> contains the amount of tez to be transferred (excluding baking fees).
<code>transfer</code> wraps the operation in an <code>annotated_manager_operation</code> by function <code>prepare_manager_operation</code>, adding fee and <a href="https://tezos.gitlab.io/alpha/glossary.html#gas">gas</a> information; then it further wraps the operation into the <span class="caps">GADT</span> <code>Injection.Single_manager</code> of type <code>annotated_manager_operation_list</code>; and finally it passes this all on to function <code>inject_manager_operation</code>, detailed below.</p>
</li>
</ul>
<p><a id="manager_operation"></a></p>
<blockquote>
<p>Manager operations are operations that compete between themselves for inclusion in a block. This is in contrast to other types of operations such as consensus or governance related operations. The competition is realized through the proposed fees, which go to the baker selecting the operations to be included in the block. There are four types of manager operations, defined by
the type <a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/operation_repr.ml#L123"><code>manager_operation</code></a> in <a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/operation_repr.ml"><code>operation_repr.ml</code></a> using the following four constructors:</p>
<ol>
<li><code>Reveal</code> for the revelation of a public key, a one-time prerequisite to any signed operation, in order to be able to check the sender’s signature.</li>
<li><code>Transaction</code> of some amount to some destination contract.</li>
<li><code>Origination</code> of a contract using a smart-contract <code>script</code> and intially credited with the amount <code>credit</code>.</li>
<li><code>Delegation</code> to some staking contract (designated by its public key hash).</li>
</ol>
<p>Although we usually use <code>tezos-client</code> to inject a single operation, the economic protocol use a built-in notion of a list of operations (also called “packed operation”) that can be injected as a whole, for efficiency reasons.
In this case the constructor <code>Injection.Cons_manager</code>, instead of <code>Injection.Single_manager</code>, would be used to build a value of type <code>annotated_manager_operation_list</code>.</p>
</blockquote>
<h2 id="step-2-injecting-the-operation"><a class="toclink" href="#step-2-injecting-the-operation">Step 2: Injecting the operation</a></h2>
<p><img alt="" src="images/proto-controlflow-transfer/tezos-transfer-tokens-src-injection.svg"></p>
<ul>
<li>
<p>The function <strong><code>inject_manager_operation</code></strong> in <code>injection.ml</code> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_client/injection.ml#L911">l.911</a>) proceeds, from the previous list of annotated manager operations (of type <code>annotated_manager_operation_list</code>), by creating a list of contents (of type <code>contents_list</code>) where each content (of type <code>contents</code>) is a <span class="caps">GADT</span> <code>Manager_operation</code>. This mapping is performed by the local function <code>build_contents</code>, and for each element by the local function <code>contents_of_manager</code>. These functions deconstruct and reconstruct the list of annotated manager operations, providing information for the missing values (fee, gas limit, storage limit).</p>
<p>Then, function <code>inject_manager_operation</code> proceeds according to two cases:</p>
<ol>
<li>If Alice’s public key has not yet been revealed, then the function adds one more manager operation to reveal her public key.</li>
<li>If Alice’s public key has been revealed, then the next function is directly called with this list of contents, which is the case in our example.</li>
</ol>
</li>
<li>
<p>The function <strong><code>inject_operation</code></strong> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_client/injection.ml#L773">l.773</a>) completes the final steps of injecting into the node. It:</p>
<ul>
<li>computes estimated gas and storage (in function <code>may_patch_limits</code>),</li>
<li>checks compliance with cap fees (in function <code>may_patch_limits</code>),</li>
<li>performs a simulation (by calling <code>preapply</code>), and finally</li>
<li>injects the operation (by calling <code>Shell_services.Injection.operation</code> in <a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_client/injection.ml#L825">l.825</a>).</li>
</ul>
</li>
</ul>
<h2 id="step-3-dispatching-the-operation"><a class="toclink" href="#step-3-dispatching-the-operation">Step 3: Dispatching the operation</a></h2>
<p>The calls to <code>Alpha_block_services.Helpers.Preapply.operations</code> in <code>preapply</code> for simulation and to <code>Shell_services.Injection.operation</code> for injection from Step 2 actually represent two <span class="caps">RPC</span> calls from the client to the node.
Such RPCs are declared in <code>src/lib_shell_services</code> and implemented in <code>src/lib_shell/</code> (in files with the name of the form <code>*_directory.ml</code>).
To serve these two RPCs, the shell part of the node simply dispatches the operation execution to the relevant economic protocol, here <code>Alpha</code>.</p>
<h2 id="step-4-applying-the-operation-in-the-economic-protocol"><a class="toclink" href="#step-4-applying-the-operation-in-the-economic-protocol">Step 4: Applying the operation in the economic protocol</a></h2>
<p><img alt="" src="images/proto-controlflow-transfer/tezos-transfer-tokens-src-protocol.svg"></p>
<ul>
<li>The main entrypoint to apply the operation which we injected in the previous section is the function <strong><code>apply_operation</code></strong> in <code>src/proto_alpha/lib_protocol/main.ml</code> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/main.ml#L180">l.180</a>). A variable <code>operation</code> (of type <code>'kind operation'</code>) containing all information for the operation itself is passed to the next function.</li>
</ul>
<blockquote>
<p>The record-type <code>'kind operation</code> defined in file <code>operation_repr.ml</code> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/operation_repr.ml#L62">l.62</a>) contains a field <code>protocol_data</code> of type <code>'kind protocol_data</code> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/operation_repr.ml#L67">l.67</a>). It wraps together a list of contents (of type <code>contents_list</code>) and a signature of the source contract for authentication purposes.</p>
</blockquote>
<ul>
<li>Next, the function <strong><code>apply_operation</code></strong> in <code>src/proto_alpha/lib_protocol/apply.ml</code> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/apply.ml#L1339">l.1339</a>) is called, which after some initialization, uses <code>operation.protocol_data.contents</code> (of type <code>contents_list</code>) to call the next function.</li>
<li>In the function <strong><code>apply_contents_list</code></strong> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/apply.ml#L1103">l.1103</a>), two function calls are worth mentioning:<ol>
<li>On one side (left in the figure), the function <strong><code>precheck_manager_contents_list</code></strong> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/apply.ml#L1327">l.1327</a>), which in turn calls the function <code>precheck_manager_contents</code> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/apply.ml#L928">l.928</a>), and itself will call <code>Contract.spend</code> will trigger the payment of baking fees (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/apply.ml#L833">l.833</a>).</li>
<li>On the other side (right in the figure), the function <strong><code>apply_manager_contents_list</code></strong> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/apply.ml#L1331">l.1331</a>) makes successive nested calls to <code>apply_manager_contents_list_rec</code> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/apply.ml#L987">l.987</a>), then <code>apply_manager_contents</code> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/apply.ml#L836">l.836</a>), and finally <code>apply_manager_operation_content</code> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/apply.ml#L518">l.518</a>). Finally, the amount of ꜩ10 is debited from Alice when calling <code>Contract.spend</code> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/apply.ml#L552">l.552</a>) and then credited to Bob when calling <code>Contract.credit</code> (<a href="https://gitlab.com/tezos/tezos/-/blob/6102c808a21b32e732ab9bb1825761cd056f3e86/src/proto_alpha/lib_protocol/apply.ml#L570">l.570</a>).</li>
</ol>
</li>
</ul>
<h1 id="summary"><a class="toclink" href="#summary">Summary</a></h1>
<p>To show how operations are handled in Tezos, we started from the command-line level using <code>tezos-client</code> to trigger a token transfer between two accounts.
This client process occurs in the <em>shell</em>, outside of the procotol.
The command itself is parsed and a manager operation is prepared.
This operation is further wrapped up as an annotated manager operation in a single-item list of the one operation.</p>
<p>Furthermore, to prepare for the injection into the <a href="https://tezos.gitlab.io/alpha/glossary.html#economic-protocol">economic protocol</a>, this list of annotated manager operations is transformed into a list of contents.
A content is a wrapper for various operations, one of them is the manager operation.
Finally, the operation is injected in <code>injection.ml</code>.</p>
<p>On the economic protocol side, the operation has been fetched, consisting of a signature and a protocol data, wrapped together with a shell header.
This value is then sent to <code>apply.ml</code> which is in charge of calling the final low-level operations for funds debit/credit.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:revelation">
<p>See <a href="https://tezos.gitlab.io/introduction/howtouse.html#transfers-and-receipts">this short tutorial</a> (search for “first a revelation”).
By default (to save space, and to provide a little additional security) only <em>hashes</em> of public keys are recorded on the blockchain.
But in order for Alice to sign a transaction from her account to Bob’s, Alice has to pay a one-time cost — ꜩ0.001259, in the example linked to in the tutorial above — for the full public key of her account to be recorded on the blockchain as a special <em>key revelation</em> operation. <a class="footnote-backref" href="#fnref:revelation" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:commands">
<p>You can fetch a list of them at the command line with <code>tezos-client man</code>. <br> We use here a bespoke command-line argument parser developed in Nomadic Labs called <a href="https://tezos.gitlab.io/api/odoc/_html/tezos-clic/Tezos_clic/Clic/index.html">Clic</a>.
Other parts of the codebase use other parsers, including <code>Arg</code>, and <a href="https://erratique.ch/software/cmdliner"><code>Cmdliner</code></a> (used in <code>tezos-node</code>). <a class="footnote-backref" href="#fnref:commands" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>Edo, the latest Tezos upgrade, is LIVE2021-02-13T20:00:00+01:002021-02-13T20:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2021-02-13:/edo-the-latest-tezos-upgrade-is-live.html<p>Summary: </p>
<p><em>This is a joint announcement from Nomadic Labs, Marigold, and DaiLambda.</em></p>
<p>On 13 February 2021, the Tezos blockchain successfully upgraded by adopting Edo at block 1,343,489. Jointly developed by Nomadic Labs, Marigold, DaiLambda, and Metastate, Edo is the fifth Tezos upgrade in the span of two years …</p><p>Summary: </p>
<p><em>This is a joint announcement from Nomadic Labs, Marigold, and DaiLambda.</em></p>
<p>On 13 February 2021, the Tezos blockchain successfully upgraded by adopting Edo at block 1,343,489. Jointly developed by Nomadic Labs, Marigold, DaiLambda, and Metastate, Edo is the fifth Tezos upgrade in the span of two years, and follows the <a href="https://decrypt.co/48019/tezos-delphi-upgrade-slashes-gas-costs-by-75">Delphi upgrade</a> of three months ago.</p>
<p>The Tezos blockchain currently allows protocol upgrades every several months, and <a href="https://forum.tezosagora.org/t/regular-scheduling-for-our-tezos-proposals/2331">we intend for the foreseeable future</a> to take advantage of every such opportunity, rapidly incorporating the best available technology into Tezos. Most cryptocurrency networks lack a mechanism to decide on the content of technical upgrades. By contrast, Tezos’ on-chain self-governance and self-amendment mechanisms allow it to evolve in a way that respects the expressed preferences of its users. </p>
<p>A full list of the changes in Edo can be found on <a href="https://tezos.gitlab.io/protocols/008_edo.html">this documentation page</a>. To summarize, however, the upgrade contains some minor bug fixes, some improvements to performance and gas consumption, the addition of a new period (named the “Adoption Period”) to the upgrade process, and two important new features that we have been working on for some time: Sapling, and Tickets.</p>
<p>Sapling is a protocol originally developed by the Electric Coin Company. Edo allows smart contract developers to easily integrate Sapling in their smart contracts, enabling new types of applications such as voting or supporting asset transactions with selective disclosures.</p>
<p>Tickets are another substantial improvement in Tezos. <strong>Tickets are currently experimental and should not be used in mission-critical contracts until we have completed our ongoing audit of its implementation.</strong> Tickets are a convenient mechanism for smart contracts to grant portable permissions to other smart contracts or to issue tokens. While it’s possible to achieve this with existing programming patterns, tickets make it much easier for developers to write secure and composable contracts, and we expect to see extensive use of tickets after the feature has stabilized.</p>
<p>Just like any other feature of the protocol, Tezos protocol amendments may make changes to the amendment process itself. The “Adoption Period” (sometimes referred to as the “Fifth Period”) is an important improvement to the governance mechanism we have wanted to make for some time.</p>
<p>Up until now, new versions of the protocol have gone live (that is, have been “activated”) one block after voting has been completed, which in practice is only sixty seconds. This has made it difficult for some Tezos bakers, indexers, and other users of the network to assure seamless upgrades of their nodes. We have also seen instances where the lack of certainty about whether an upgrade would be adopted has caused some users to delay preparations until the last moment.</p>
<p>Under the new system, instead of four periods of eight cycles during voting, the Tezos upgrade process now lasts five periods of five cycles. The new fifth period, the Adoption Period, will be a five cycle (approximately two weeks) gap between the acceptance of the new protocol in the Promotion Vote Period, and the time when it is activated on the Tezos network. This will aid in assuring seamless protocol transitions.</p>
<p>We are exceptionally pleased with the progress that has been made on the Tezos protocol and its software in recent months. Thank you to all the developers who worked so hard to make Edo a reality. We intend to inject a new proposal, “Florence”, with a variety of interesting new features, some small, some large, within a few weeks.</p>IMPORTANT: Critical Patch to Tickets in Edo2021-02-10T23:45:00+01:002021-02-10T23:45:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2021-02-10:/important-critical-patch-to-tickets-in-edo.html<p>Summary:</p>
<p>We have discovered a critical bug within the new Tickets functionality
in Edo.</p>
<p>Several mechanisms were considered to mitigate this problem;
none were ultimately found to be satisfactory. We have therefore taken
the step of producing and releasing version 8.2 of the Tezos node that
includes a patched …</p><p>Summary:</p>
<p>We have discovered a critical bug within the new Tickets functionality
in Edo.</p>
<p>Several mechanisms were considered to mitigate this problem;
none were ultimately found to be satisfactory. We have therefore taken
the step of producing and releasing version 8.2 of the Tezos node that
includes a patched version of the Edo protocol that differs by only a
few lines of code.</p>
<p>Nodes running 8.2 will automatically adopt the
patched version rather than the original version of Edo when it
activates on February 13th, 2021, around 19:30 <span class="caps">GMT</span>. We ask all bakers
and node operators to please update immediately to 8.2, rather than
8.1 which most are currently running. Nodes running version 8.1 or
earlier will not be able to communicate with the new chain.</p>
<p>The hash of the new version of Edo is:
<code>PtEdo2ZkT9oKpimTah6x2embF25oss54njMuPzkJTEi5RqfdZFA</code></p>
<p>We appreciate that this is coming at a very late stage in the Edo
upgrade process, but we believe that this is the best choice
available. It is safer to have the upgrade occur at a time when every
bakeries already updating their software and planning to pay close
attention to their node because of the impending adoption of Edo.</p>
<p>We do our best to prevent bugs from slipping inside our releases but
unfortunately, as with any complex software, there is always a small
chance of missing something. We intend to adopt several new quality
control mechanisms to reduce the probability of similar bugs going
undetected in the future.</p>
<hr>
<p>The Git tag for this release is <code>v8.2</code> and the corresponding commit hash
is <code>6102c808a21b32e732ab9bb1825761cd056f3e86</code>.</p>
<p>Full changelog and update instructions are available in the version 8 <a href="https://tezos.gitlab.io/releases/version-8.html">release page</a>.</p>A look ahead to Tenderbake2021-02-08T12:00:00+01:002021-02-08T12:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2021-02-08:/a-look-ahead-to-tenderbake.html<p>We’re working on changing the Tezos consensus algorithm
from the current <a href="https://tezos.gitlab.io/007/proof_of_stake.html">Emmy<sup>+</sup></a> algorithm,
to a new algorithm called <a href="https://arxiv.org/abs/2001.11965">Tenderbake</a>.
We’d like to discuss this development, and explain why we’re considering it, and what advantages it will bring.</p>
<p>Tenderbake and Emmy<sup>+</sup> belong to different algorithm families:</p>
<ul>
<li>Emmy<sup>+</sup> is a <em><a href="http://p2pfoundation.ning.com/forum/topics/bitcoin-open-source">Nakamoto style</a></em> algorithm, whereas</li>
<li>Tenderbake is a <em><a href="https://lamport.azurewebsites.net/pubs/byz.pdf">classical <span class="caps">BFT</span>-style</a></em> algorithm.</li>
</ul>
<p>So moving to Tenderbake would be a significant development on the Tezos network.</p>
<p>We made every effort to keep this blog post self-contained, but just in case, you might also find this <a href="https://tezos.gitlab.io/007/glossary.html">glossary of the technical terms</a> useful!</p>
<h2 id="why-tenderbake"><a class="toclink" href="#why-tenderbake">Why Tenderbake?</a></h2>
<h3 id="tenderbake-has-quick-deterministic-finality"><a class="toclink" href="#tenderbake-has-quick-deterministic-finality">Tenderbake has quick, deterministic finality</a></h3>
<p>From the point of view of the user, Tenderbake’s killer feature, relative to Emmy<sup>+</sup>, is that it offers <em>deterministic finality</em>: a block that has just been appended to the chain of some node is known to be <em>final</em> once it has two additional blocks on top of it, regardless of network latency.</p>
<p>Tenderbake is also fast, in the sense that it has a small (quick) time to finality: under typical good
network conditions, and making a standard assumption of an attacker (“byzantine”) stake of at most 33% — meaning that at most 1/3 of the network is trying to undermine correct behaviour:</p>
<p><a name="estim"></a></p>
<ul>
<li>in Tenderbake, one would expect to wait less than 1 minute for a block to be considered final, whereas</li>
<li>in Emmy<sup>+</sup>, one would expect to wait at least <a href="/emmy-in-the-partial-synchrony-model.html#forks-started-in-the-past">6 minutes</a>.</li>
</ul>
<p>How this estimate is made will become clearer <a href="#rd">below</a>, when we introduce the notion of round duration.</p>
<p>We now expand on some of these claims:</p>
<h4 id="probabilistic-finality-nakamoto-style"><a class="toclink" href="#probabilistic-finality-nakamoto-style">Probabilistic finality (Nakamoto style)</a></h4>
<p>Blockchains are decentralized, so consensus must likewise be decentralized.
So suppose there exist blocks <span class="math">\(b\)</span>, <span class="math">\(b_1\)</span>, and <span class="math">\(b_2\)</span> such that some participants on the blockchain’s network believe that <span class="math">\(b_1\)</span> immediately follows <span class="math">\(b\)</span>, whereas others believe that <span class="math">\(b_2\)</span> immediately follows <span class="math">\(b\)</span>:
there is no central authority to enforce consensus on which participants are “right” and which are “wrong”, and we say the blockchain is <strong>forked</strong>, or in a <strong>forked state</strong>.</p>
<p>In Emmy<sup>+</sup>, like in all Nakamoto-style consensus algorithms, forks can have arbitrary length. However, forked states become exponentially unstable and tend to collapse down to a single branch (assuming <a href="/emmy-in-the-partial-synchrony-model.html">decent bounds on network latency</a>).</p>
<p>When a fork collapses to a single branch, we say that its blocks have reached <em>finality</em>. So, we say that Emmy<sup>+</sup> has <strong>probabilistic finality</strong> because forks of arbitrary length are possible but they collapse with probability that increases suitably rapidly with fork length.</p>
<h4 id="deterministic-finality-classical-bft-style"><a class="toclink" href="#deterministic-finality-classical-bft-style">Deterministic finality (classical <span class="caps">BFT</span>-style)</a></h4>
<p>In Tenderbake, forks are impossible.
Or to put it slightly differently: forks collapse down to finality <em>after just two blocks</em>, always and regardless of network latency. This is <strong>deterministic finality</strong>.
How this is achieved, is the topic of this post.</p>
<p><a name="twoblks"></a>
We need two blocks because:</p>
<ol>
<li>The head of the blockchain is a candidate block to be agreed upon, and</li>
<li>its parent is a block whose non-consensus operations have been agreed, but the consensus operations might still change (on rare occasions).</li>
</ol>
<h3 id="tenderbake-is-safe-under-asynchrony"><a class="toclink" href="#tenderbake-is-safe-under-asynchrony">Tenderbake is safe under asynchrony</a></h3>
<p>From the point of view of a security analyst, Tenderbake has an advantage over Emmy<sup>+</sup>: no fork is possible, regardless of network delays, even during an <a href="https://decentralizedthoughts.github.io/2019-06-01-2019-5-31-models/">asynchronous</a> period.</p>
<p><a name="partial-synchrony"></a>
Let’s explain. Our network model assumes <strong>partial synchrony</strong>, which splits time into two kinds of periods:</p>
<ul>
<li>During a <em>synchronous</em> period, there’s a global bound on the delivery delay for any network message.
<br> This is (hopefully) the usual state of affairs. Messages are delivered promptly (for a certain global value of “promptly”).</li>
<li>During an <em>asynchronous</em> period, network performance may be degraded: but, every asynchronous period is finite.
<br> This is (hopefully) an exceptional state of affairs.</li>
</ul>
<p>This is a realistic scenario because in the real world,</p>
<ul>
<li>networks work …</li>
<li>… until they don’t, and then action is taken to fix the problem and service is restored in finite time.</li>
</ul>
<p>So it is important for a consensus algorithm to</p>
<ul>
<li>function efficiently during synchrony, and just as important to</li>
<li>(perhaps degrade but then) recover gracefully from asynchrony.</li>
</ul>
<p>So back to our security analyst: Tenderbake guarantees that even if the network degrades during an asynchronous period, once synchrony is restored our blockchain will just pick up where it left off.
Nakamoto-style consensus does not have this guarantee, and may emerge from an asynchronous period having developed long forks.</p>
<p><a name="live"></a></p>
<h3 id="nakamoto-is-live-classical-bft-is-safe"><a class="toclink" href="#nakamoto-is-live-classical-bft-is-safe">Nakamoto is “live”, classical <span class="caps">BFT</span> is “safe”.</a></h3>
<p>As is so often the case in consensus, design boils down to making trade-offs: given that some asynchronous periods will eventually occur, do we prefer to be “live” or “safe” during them?</p>
<p>Nakamoto-style consensus algorithms favour being <em>live</em>: blocks are always produced, even in an asynchronous period.
Classical <span class="caps">BFT</span>-style algorithms favour being <em>safe</em>: production of blocks pauses during an asynchronous period.</p>
<p>Our network model when designing Tenderbake assumes that any period of asynchrony is finite — we assume that somebody is going to rush to fix the broken cable, protect against the DoS attack, and/or bring the server back online. So it makes sense to be safe and (essentially) just wait for the network to come back online.</p>
<h2 id="from-tendermint-to-tenderbake-a-journey"><a class="toclink" href="#from-tendermint-to-tenderbake-a-journey">From Tendermint to Tenderbake: a journey</a></h2>
<h3 id="tendermint"><a class="toclink" href="#tendermint">Tendermint</a></h3>
<p>The starting point of our work was
<a href="https://arxiv.org/abs/1807.04938">Tendermint</a>, one of the first classical <span class="caps">BFT</span>-like blockchain algorithms.</p>
<p>Tenderbake is ‘just’ a version of Tendermint adapted for the Tezos blockchain,
but the adjustments required are substantive, as we discuss below. In summary:</p>
<ul>
<li>Tenderbake is tailored to match the Tezos architecture by using only communication primitives and network assumptions which Tezos supports.</li>
<li>In particular, Tenderbake makes <a href="#wa">weaker network assumptions than Tendermint</a>, at the price of adding the extra assumption that participants have <em>loosely synchronized clocks</em><sup id="fnref:lsc"><a class="footnote-ref" href="#fn:lsc">1</a></sup> — which is fine, because Tezos uses them.</li>
</ul>
<h3 id="the-tezos-architecture"><a class="toclink" href="#the-tezos-architecture">The Tezos architecture</a></h3>
<p>When we adapted Tendermint consensus to the Tezos architecture to arrive at Tenderbake, we had to account for the following structural features of Tezos:</p>
<ol>
<li>
<p>Tezos’ self-amending feature means there is a separation between</p>
<ul>
<li>the <strong>shell</strong>, which handles low-level communication, and</li>
<li>the <strong>economic protocol</strong>, which actually does the blockchain stuff and which can be updated by a Tezos amendment.</li>
</ul>
<p>The Tezos shell is for our purposes immutable, so our Tenderbake implementation has to fit into the more easily-updated economic protocol. In particular, we can use only communication primitives supported by the Tezos shell.</p>
</li>
<li>
<p>Furthermore, Tezos distinguishes between</p>
<ul>
<li>a <strong>node</strong>, which is on the peer-to-peer network and whose job is to manage communication with other nodes and the validation and the storage of blocks, and</li>
<li>a <strong>baker</strong>, who produces blocks and communicates with just one node.</li>
</ul>
<p>We recall that <a href="https://tezos.gitlab.io/007/glossary.html#baker">bakers</a> hold <a href="https://tezos.gitlab.io/007/glossary.html#delegate">delegates’</a> keys and therefore for safety they are <strong>sandboxed</strong> — not directly exposed to the peer-to-peer network. So nodes manage communication and shield bakers from network attack, and bakers hold secrets and bake blocks into the blockchain. This arrangement affects what network assumptions we can make.</p>
</li>
</ol>
<p>So, treating these points in turn:</p>
<ol>
<li>
<p>Tendermint uses three types of consensus messages: <em>proposals</em>, <em>prevotes</em>, and <em>precommits</em>. We must map these to the following available shell communication primitives:</p>
<ul>
<li>an <em>operation</em>; and</li>
<li>a <em>block</em>, which consists of<ul>
<li>some <em>block contents</em> which is a list of operations, and</li>
<li>a <em>block header</em> which contains a hash of the block contents (for efficiency and as a checksum for data integrity).<sup id="fnref:header"><a class="footnote-ref" href="#fn:header">4</a></sup></li>
</ul>
</li>
</ul>
<p>We choose as follows: <a name="blk_prop"></a></p>
<ul>
<li>Tenderbake maps Tendermint <em>proposals</em> to the native Tezos action of proposing Tezos blocks, and</li>
<li>Tenderbake maps Tendermint <em>prevotes</em> and <em>precommits</em> to Tezos <a href="https://tezos.gitlab.io/007/glossary.html#operations">operations</a>.<sup id="fnref:operation"><a class="footnote-ref" href="#fn:operation">5</a></sup></li>
</ul>
</li>
<li>
<p>Bakers are semi-isolated from the network for their own safety, as discussed above, so Tenderbake must weaken the network assumptions of Tendermint<sup id="fnref:tendermintna"><a class="footnote-ref" href="#fn:tendermintna">3</a></sup> because those assumptions are too strong to be guaranteed by the Tezos peer-to-peer layer.</p>
<p><a name="wa"></a>
Tendermint assumes <strong>reliable broadcast</strong>: if a correct validator receives a message then all correct validators receive it. Notably, this requires that messages may be delayed, but messages may not be lost, during an asynchronous period.</p>
<p>Tenderbake does not assume communication is reliable. Messages sent during an asynchronous period might never arrive. In fact, Tenderbake is designed to work without any additional assumption except partial synchrony. The technical jargon is: it assumes a best-effort broadcast primitive. <strong>Best-effort broadcast</strong> means that if a correct validator sends a message during a synchronous period, then all correct (non-byzantine) validators receive it. Best-effort broadcast is the broadcast primitive implicit in the <a href="#partial-synchrony">definition of partial synchrony above</a>.</p>
</li>
</ol>
<h3 id="the-tenderbake-consensus-algorithm-in-brief"><a class="toclink" href="#the-tenderbake-consensus-algorithm-in-brief">The Tenderbake consensus algorithm in brief</a></h3>
<h4 id="levels"><a class="toclink" href="#levels">Levels</a></h4>
<p>Some terminology:</p>
<ul>
<li>The <strong>level</strong> of a block is the number of blocks since the genesis block, where the genesis block is at level 0.<sup id="fnref:1984"><a class="footnote-ref" href="#fn:1984">7</a></sup></li>
<li>The <em>fundamental unit of identity</em> in Tezos is a cryptographic key. A <strong>delegate</strong> is then a cryptographic key whose owner can participate in the consensus algorithm and in the governance process by virtue of registering (the public part of) their key on the chain, holding at least one <a href="https://tezos.gitlab.io/007/proof_of_stake.html#rolls">roll</a> (= 8,000 tez), and having been active recently.</li>
</ul>
<p>Tenderbake is executed for each new block level by <strong>validators</strong>, which are delegates selected at random based on their stake,<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">8</a></sup> in the same way as endorsers <a href="https://tezos.gitlab.io/007/proof_of_stake.html#endorsements">are selected in Emmy<sup>+</sup></a>.</p>
<p>The validators’ task is to agree on which block to add next.
In Tendermint, this process is</p>
<ul>
<li>started by each validator emitting a <em>proposal</em> message, which is just an abstract message in the algorithm proposing a block, and</li>
<li>continued by validators voting on which proposal to accept, by a voting mechanism which we will describe in more details shortly; what matters for now is that voting takes place so voting messages must be communicated.</li>
</ul>
<p>Tenderbake has to choose a concrete representation for this process.
The natural way to propose a block is for a validator to use the native Tezos proposal mechanism to <em>actually propose</em> its preferred block as the next block to add to the next level in the Tezos blockchain.
Validators can then vote on the proposed blocks by communicating their votes across the Tezos network as <em>Tezos operations</em>.<sup id="fnref:emmy"><a class="footnote-ref" href="#fn:emmy">9</a></sup></p>
<p>So schematically, Tenderbake acts as follows:</p>
<ol>
<li>a validator injects a candidate block (representing a <em>proposal</em>) and consensus operations (representing <em>votes</em>) into the node to which it is attached, which then</li>
<li>diffuses those blocks and consensus operations to other nodes of the network, and thus</li>
<li>communicates them to the validators attached to those nodes,</li>
<li>to carry out voting on which block to accept.</li>
</ol>
<p>We now consider how the voting process works, in more detail.</p>
<p><a name="rd"></a></p>
<h4 id="levels-are-composed-of-rounds"><a class="toclink" href="#levels-are-composed-of-rounds">Levels are composed of rounds</a></h4>
<p>For each level, Tenderbake proceeds in <strong>rounds</strong>.
Each round represents an attempt by the validators to agree on some block for the current level.</p>
<p>Each round has an associated <strong>duration</strong>.
Round durations are set to increase so that for any possible message delay and/or asynchronous period (when the network may be slow or unreliable), there is a round that is longer.</p>
<p>Each round has three phases:</p>
<ol>
<li>a <strong>block proposal</strong> phase;</li>
<li>a <strong>preendorsement</strong> voting phase, and</li>
<li>an <strong>endorsement</strong> voting phase.</li>
</ol>
<p>In more detail:</p>
<ol>
<li>In the <em>block proposal</em> phase, one of the validators is designated as the round’s <strong>proposer</strong>. The proposer’s task is to propose a <strong>candidate block</strong> (more on how this is chosen in a moment).</li>
<li>In the <em>preendorsement</em> phase, validators send a supporting vote (a preendorsement) on the candidate block’s contents. So, this is a vote in support of the non-consensus operations<sup id="fnref:precise"><a class="footnote-ref" href="#fn:precise">10</a></sup> that it contains. <br> Concretely, a preendorsement is a Tezos operation containing a tuple <span class="math">\((x,bc,r)\)</span> with intended semantics <br> <em>“I, validator <span class="math">\(x\)</span>, hereby endorse block contents <span class="math">\(bc\)</span> for round <span class="math">\(r\)</span><span class="dquo">“</span>.</em></li>
<li>In the <em>endorsement</em> phase, provided validators have observed a <em>quorum</em><sup id="fnref:quorum"><a class="footnote-ref" href="#fn:quorum">11</a></sup> of preendorsements for the block contents, validators send a confirmation vote (an endorsement) for the contents of the candidate block and for the preendorsement quorum. <br> Concretely, an endorsement has the intended semantics <br> <em>“I, validator <span class="math">\(x\)</span>, hereby affirm to have observed a quorum of preendorsemeents for block contents <span class="math">\(bc\)</span> at round <span class="math">\(r\)</span><span class="dquo">“</span>.</em></li>
</ol>
<h4 id="first-round-vs-subsequent-rounds"><a class="toclink" href="#first-round-vs-subsequent-rounds">First round vs. subsequent rounds</a></h4>
<p>In the first round, the proposer is free to propose some block contents taken from its node’s mempool — this being a pool of pending operations that the node has accumulated, but which have not yet been baked into a block on the blockchain.
In subsequent rounds, the proposer must propose the contents of the block candidate from the previous round, <em>provided</em> a quorum of preendorsements was observed for the block contents proposed in the previous round; otherwise, the proposer is free to choose from its node’s mempool, as for the first round.</p>
<p>If a validator observes an endorsement quorum at its current round for that round’s candidate block, then the validator considers that agreement has been reached on the block candidate’s contents for level <span class="math">\(\ell\)</span>,<sup id="fnref:proof"><a class="footnote-ref" href="#fn:proof">12</a></sup> and the validator can move to level <span class="math">\(\ell+1\)</span>.
Each block candidate at level <span class="math">\(\ell+1\)</span> will have to include the endorsement quorums of level <span class="math">\(\ell\)</span>, as a proof that agreement was indeed reached at level <span class="math">\(\ell\)</span>.<sup id="fnref:signature"><a class="footnote-ref" href="#fn:signature">13</a></sup></p>
<p>If a validator does not observe an endorsement quorum at its current round for that round’s candidate block, then the validator considers that agreement has not been reached, and the validator loops back into a next round.
This may be caused by:</p>
<ul>
<li>a byzantine proposer which does not make any proposal, or proposes two candidate blocks, or</li>
<li>network delays: (pre)endorsements are sent but just don’t arrive in time, so that reaching a quorum times out.</li>
</ul>
<h4 id="a-final-round"><a class="toclink" href="#a-final-round">A final round</a></h4>
<p>Eventually there will be a round with</p>
<ul>
<li>a correct (non-byzantine) proposer who proposes precisely one candidate block as required, and</li>
<li>the round is long enough so that any asynchronous periods have passed and network messages arrive in time, so that</li>
<li>a quorum is observed and a candidate block for that round is agreed upon,<sup id="fnref:progress"><a class="footnote-ref" href="#fn:progress">15</a></sup></li>
</ul>
<p>and our loop terminates with consensus.<sup id="fnref:caveat"><a class="footnote-ref" href="#fn:caveat">14</a></sup></p>
<h4 id="how-long-does-this-all-take"><a class="toclink" href="#how-long-does-this-all-take">How long does this all take?</a></h4>
<p>Assuming good network conditions, validators will agree on a block after just one round, so that each level lasts one round.
Also, <a href="#twoblks">finality is in two blocks</a>, so that finality is achieved after two levels = two blocks.<sup id="fnref:tendermint-immediate"><a class="footnote-ref" href="#fn:tendermint-immediate">16</a></sup>
Thus, assuming good network conditions, the estimated time for block finality is one minute at most.</p>
<p>On first experiments on a private testnet for Tenderbake, the duration of the first round was set to 15s, so we confirmed that this one-minute estimate is experimentally accurate.</p>
<h4 id="why-do-we-need-preendorsements"><a class="toclink" href="#why-do-we-need-preendorsements">Why do we need (pre)endorsements?</a></h4>
<p>Tenderbake has two voting phases: preendorsements and endorsements. Why vote twice?</p>
<p>These algorithms are genuinely subtle, and this particular design is typical of classical <span class="caps">BFT</span>-style consensus algorithms, going back for example to <a href="https://groups.csail.mit.edu/tds/papers/Lynch/podc84-DLS.pdf">the seminal 1984 <span class="caps">DLS</span> (Dwork-Lynch-Stockmeyer) paper</a>.
We offer intuition as to why having two phases can be helpful, which — to be clear — is not intended as a definitive explanation:</p>
<p>Intuitively, having two phases helps to ensure agreement and progress even if the network gets partitioned or participants crash.</p>
<p>Suppose we had only <em>one</em> voting phase (so no preendorsements).
Then either we allow validators to change their vote in later rounds, or we don’t:</p>
<ul>
<li><em>If we allow validators to change vote in later rounds…</em> <br>
… then suppose only one validator <span class="math">\(v\)</span> sees an endorsement quorum and decides on some block <span class="math">\(b\)</span>, but then it crashes.
The other validators are not aware of <span class="math">\(v\)</span><span class="quo">‘</span>s decision, so they may later vote on a different block.
Thus, the agreement property may be broken.</li>
<li><em>If we don’t allow validators to change vote in later rounds…</em> <br>
… then suppose the proposer for a round is Byzantine and it proposes two blocks <span class="math">\(b\)</span> and <span class="math">\(b'\)</span>; then some validators vote on <span class="math">\(b\)</span>, and some vote on <span class="math">\(b'\)</span>, and neither <span class="math">\(b\)</span> nor <span class="math">\(b'\)</span> gathers a quorum of votes.
Then there will be no agreement, because validators have to stick with the votes forever. Thus, the progress property is broken.</li>
</ul>
<p>This is solved by using two voting phases.
Indeed, with <em>two</em> voting phases, participants only endorse a block once they are confident that a preendorsement quorum already exists for it. This adds stability: a fragmented network might delay consensus, but it cannot enable a fragmented consensus.</p>
<p>To quote Pierre Chambart, answering the question “Why preendorsements?”:</p>
<blockquote>
<p>Ça permet de mesurer si tu es dans la majorité, avant de prendre la décision de voter pour de vrai. (J’imagine la gueule d’une vraie élection si les gens ne voulaient voter que pour le vainqueur).</p>
<p>This allows you to know you’re voting with the majority, before casting your final vote. (Imagine what a real election would look like, if people only wanted to vote for the winner.)</p>
</blockquote>
<p>The interested reader can also watch Ittai Abraham’s excellent tutorials:</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=weqRVt5OwNY&feature=youtu.be">Byzantine fault tolerance, state machine replication and blockchains</a> and</li>
<li><a href="https://www.youtube.com/watch?v=ONobI3X70Rc&feature=youtu.be">The HotStuff approach to <span class="caps">BFT</span></a> (Hotstuff is a successor to Tendermint).</li>
</ul>
<h2 id="where-are-we"><a class="toclink" href="#where-are-we">Where are we?</a></h2>
<ol>
<li>We proved on <a href="https://arxiv.org/abs/2001.11965">paper</a> that Tenderbake is correct, in a sense made formal by Theorems 5 and 6 of that paper.</li>
<li>Then, we worked on a prototype to test our approach.
This prototype took the form of a <a href="https://gitlab.com/nomadic-labs/tezos/-/tree/tenderbake/src/proto_tenderbake/">“demo” economic protocol</a>,
which includes just the consensus algorithm and a basic account system (but nothing more: e.g. no rolls, delegation, smart contracts).</li>
<li>We are now close to a fully-featured economic protocol (<a href="https://gitlab.com/nomadic-labs/tezos/-/tree/tenderbake-delphi/src/proto_alpha">a modified version of Delphi using Tenderbake instead of Emmy+</a>).</li>
<li>A private testnet is running a modified version of Delphi using Tenderbake instead of Emmy+. After fixing some observed issues, we plan to make this testnet public.</li>
</ol>
<h2 id="conclusions"><a class="toclink" href="#conclusions">Conclusions</a></h2>
<p>So that’s it! Tenderbake is classical <span class="caps">BFT</span>-like whereas Emmy<sup>+</sup> is Nakamoto-like.
Emmy<sup>+</sup> has probabilistic finality making it more live but less safe, whereas Tenderbake has deterministic finality making it more safe but less live.
Tenderbake also has noticeably quicker time to finality.</p>
<p>We’re testing it now: so watch this space.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:lsc">
<p>Recall our network model assumes that we are always either: in a synchronous period, when there exists a global bound <span class="math">\(\delta\)</span> such that for every pair of nodes on the Tezos network, messages get delivered within time <span class="math">\(\delta\)</span>; or in an asynchronous period, when they don’t, but asynchronous periods only ever last finite time.
<br><br>
Say a network has <strong>loosely synchronized clocks</strong> when for every synchronous period there exists some constant <span class="math">\(\rho\)</span> such that and for any blockchain participant, the time error (also called “clock drift”; the difference between the real time and the participant’s local clock) is bounded by <span class="math">\(\rho\)</span>.
<br><br>
The values of <span class="math">\(\delta\)</span> and <span class="math">\(\rho\)</span> in the two paragraphs above are <em>a priori</em> unknown.<sup id="fnref:unknown"><a class="footnote-ref" href="#fn:unknown">2</a></sup>
<br><br>
During an asynchronous period, Tenderbake does not assume any bound on clock drift.
This is reasonable: if network delays are unbounded then it is reasonable to suppose that timing messages might be arbitrarily delayed.
Conversely, during a synchronous period when there is a global bound on message delays, we may assume that timing messages arrive after a bounded delay, so that clock drift is also bounded.
<br><br>
<em>Technical note:</em> There is a stability assumption in the implemented system, that the <a href="#rd">duration of the first round</a> must be greater than the clock drift <span class="math">\(\rho\)</span>. This guarantees that a validator can ignore all messages that do not match its current round, which ensures that message buffers remain bounded, and the validator cannot be spammed with irrelevant messages. <a class="footnote-backref" href="#fnref:lsc" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:unknown">
<p><span class="quo">‘</span>Unknown’ here means that <span class="math">\(\delta\)</span> and <span class="math">\(\rho\)</span> are parameters: an external observer of the system might be able to observe the system and calculate or deduce a value of <span class="math">\(\delta\)</span> or <span class="math">\(\rho\)</span>, but no algorithm within the system itself is allowed to depend them having any particular value.
See also <a href="https://research-development.nomadic-labs.com/emmy-in-the-partial-synchrony-model.html#fn:2">a similar discussion for partial synchrony</a>. <a class="footnote-backref" href="#fnref:unknown" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:tendermintna">
<p>See <a href="https://arxiv.org/abs/1807.04938">Section <span class="caps">II</span>.A of the Tendermint paper</a>. <a class="footnote-backref" href="#fnref:tendermintna" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:header">
<p>The block header contains other useful metadata, including: a hash of its corresponding block contents as mentioned above, a hash of its predecessor block, the level, the round, and a timestamp. This absolutely doesn’t matter for this blog post, but if you’re reading this footnote then you’re probably the kind of person who would want to know. You asked your parents to read the encyclopaedia to you at bedtime instead of “made-up stories”, too, didn’t you? <a class="footnote-backref" href="#fnref:header" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:operation">
<p>A Tezos “operation” is just the basic unit of information in Tezos.
An operation is labelled by a <em>kind</em> (called a “pass” in the source code) which is an integer such that <span class="math">\(0\)</span> indicates “this operation is a consensus message” and any other number indicates “this may be something else”.
It is up to the protocol to decide how to interpret operations — an operation is first created, then injected into some node’s mempool, transmitted on the network, and finally possibly (but not necessarily) inserted in a block.
So there is nothing particularly fancy about encoding prevotes and precommits: they are just data which can be encoded as operations<sup id="fnref:mapping"><a class="footnote-ref" href="#fn:mapping">6</a></sup> and — maybe, maybe not — baked into a block on the blockchain. <a class="footnote-backref" href="#fnref:operation" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:mapping">
<p>We could map Tendermint prevotes to blocks (instead of operations), but it would be inefficient. The algorithm requires a quorum of prevotes, so we’d need <a href="#quorum"><span class="math">\(2f+1\)</span></a> blocks to reach a quorum, where <span class="math">\(n\)</span> is the number of validators. With <span class="math">\(150\)</span> validators, prevotes would generate <span class="math">\(100\)</span> blocks just to generate consensus on the next ‘real’ block to be added.
<br><br>
It might be possible to map block proposals to operations. This was discussed, but it would require changing the shell: a proposal would have to be a special bundled operation that behaves almost like a block; when a node sees a proposal operation it would need to ask its neighbors to provide the operations contained in the bundle (the bundle operation would not actually contain operations but just pointers to, i.e. hashes of, operations). Changing the shell is anyway a big deal, and there’s clearly an easier way of just using the native Tezos block proposal mechanism, so proposals are blocks and not operations. <a class="footnote-backref" href="#fnref:mapping" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:1984">
<p>If blocks were years then a block of level 1984 would be <a href="https://en.wikipedia.org/wiki/Back_to_the_Future">the year 1985 <span class="caps">AD</span></a>.
Fun fact: our calendar goes from 1 <span class="caps">BC</span> to 1 <span class="caps">AD</span>. There is no year zero. <a class="footnote-backref" href="#fnref:1984" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>The running economic protocol makes a random choice from the available delegates (weighted by their stake) to be validators for the current block level. To be quite precise, at each level, <span class="math">\(n\)</span> <a href="https://tezos.gitlab.io/007/proof_of_stake.html#rolls">rolls</a> are selected at random and their owners are the validators for that level. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
<li id="fn:emmy">
<p><a href="https://tezos.gitlab.io/007/proof_of_stake.html#endorsements">Emmy<sup>+</sup> uses a similar mechanism here.</a> <a class="footnote-backref" href="#fnref:emmy" title="Jump back to footnote 9 in the text">↩</a></p>
</li>
<li id="fn:precise">
<p><em>Non-consensus operations</em> means operations other than preendorsements and endorsements. Notably, non-consensus operations include transactions, which for a consensus algorithm are just data, but which for the end user are the whole point of the exercise. By <em>block contents</em> we intend all operations other than preendorsements and endorsements. (Pre)endorsement operations contain the hash of the putative “block contents”, not the operations themselves. <a class="footnote-backref" href="#fnref:precise" title="Jump back to footnote 10 in the text">↩</a></p>
</li>
<li id="fn:quorum">
<p><a name="quorum"></a><em>An aside on quorums and the quorum intersection property.</em> <br> Suppose for simplicity that everybody has the same stake, so we can ignore stake-based weightings.
Suppose there are <span class="math">\(n=3f+1\)</span> validators, where <span class="math">\(f\)</span> is the number of incorrect, <a href="https://lamport.azurewebsites.net/pubs/byz.pdf">Byzantine</a> validators — the standard assumption is that at most one third of validators are Byzantines, so taking <span class="math">\(n=3f+1\)</span> is the worst case.
A <strong>quorum</strong> is a set of <span class="math">\(2f+1\)</span> signatures.
Then by the <a href="https://en.wikipedia.org/wiki/Pigeonhole_principle">pigeonhole principle</a>, quorums intersect at <span class="math">\(f+1\)</span> signatures, and given the bound <span class="math">\(f\)</span> on Byzantine actors, it follows that there is at least one signature of a correct validator.
This is called the <strong>quorum intersection property</strong>. <a class="footnote-backref" href="#fnref:quorum" title="Jump back to footnote 11 in the text">↩</a></p>
</li>
<li id="fn:proof">
<p>By the quorum intersection property, there cannot be two
endorsement quorums on two different block contents at the same
round, because then a correct validator would have endorsed two
different block contents at the same round, which is forbidden by
the algorithm. This observation is at the core of the proof that
there cannot be two different blocks agreed upon at the same
level. <a class="footnote-backref" href="#fnref:proof" title="Jump back to footnote 12 in the text">↩</a></p>
</li>
<li id="fn:signature">
<p>Anyone can check that the endorsements are produced by the right delegates, that is, by the validators for the corresponding level. This is because validators are known well in advance and their public keys are available on-chain. <a class="footnote-backref" href="#fnref:signature" title="Jump back to footnote 13 in the text">↩</a></p>
</li>
<li id="fn:caveat">
<p>For each validator, there is <em>a</em> final round, but final rounds may differ for different validators, and different validators may execute the loop a different number of times. So: rounds are a <em>per validator</em> quantity. <a class="footnote-backref" href="#fnref:caveat" title="Jump back to footnote 14 in the text">↩</a></p>
</li>
<li id="fn:progress">
<p>The property that agreement is eventually reached is expressed precisely by a <a href="https://arxiv.org/abs/2001.11965">termination property</a> — which in the context of a blockchain becomes a progress property (each level terminates, so we make progress).
An attacker could try to delay the network at a rate calculated to be <em>just a little more</em> than the increase in round durations, so that for each round <span class="math">\(r\)</span> the proposer at round <span class="math">\(r\)</span> times out, though our network model assumes this cannot go on forever: any asynchronous period must be bounded.
Following <a href="#live">our discussion above</a> of how classical <span class="caps">BFT</span>-style consensus algorithms (like Tendermint and Tenderbake) favour safety over liveness, we can draw the following chain of informal entailments: “asynchronous periods assumed bounded” <span class="math">\(\Rightarrow\)</span> <span class="dquo">“</span>we can safely err on the side of safety” <span class="math">\(\Rightarrow\)</span> <span class="dquo">“</span>deterministic finality”. <a class="footnote-backref" href="#fnref:progress" title="Jump back to footnote 15 in the text">↩</a></p>
</li>
<li id="fn:tendermint-immediate">
<p>Tendermint has <em>immediate finality</em>, meaning deterministic finality after just one block: once a block that was proposed in a proposal message is agreed upon, it is final. Tenderbake requires the marginally slower deterministic finality after two blocks because of its stronger timing assumptions: in order for validators to synchronize in their current round, proposal messages contain a round identifier, and in Tenderbake, <a href="#blk_prop">proposal messages are encoded as blocks</a>, thus blocks contain round identifiers — <em>however</em>, validators may take their decision at different rounds, so their decision rounds may differ! Thus, Tenderbake has more to agree on than Tendermint: its committee of validators must first agree on the “true” decision round to be able to agree on which is the “true” block. <a class="footnote-backref" href="#fnref:tendermint-immediate" title="Jump back to footnote 16 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>POPL 2021 retrospective2021-02-01T20:00:00+01:002021-02-01T20:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2021-02-01:/popl-2021-retrospective.html<p>A short summary of our experience at <span class="caps">POPL</span>, <span class="caps">CPP</span> and CoqPL this year.</p><!-- What's a POPL and why we care -->
<h2 id="about-popl-symposium-on-principles-of-programming-languages"><a class="toclink" href="#about-popl-symposium-on-principles-of-programming-languages">About <span class="caps">POPL</span> (Symposium on Principles of Programming Languages)</a></h2>
<p><a href="https://popl21.sigplan.org"><span class="caps">POPL</span> 2021</a> — the 48th <span class="caps">ACM</span> <span class="caps">SIGPLAN</span> Symposium on Principles of Programming Languages — is a premier annual conference event of the Programming Languages research community.
<span class="caps">POPL</span> consists of a main conference, and many colocated events, which together present the latest and greatest research in (amongst other topics):</p>
<ul>
<li>programming languages theory,</li>
<li>formal verification,</li>
<li>type systems, and</li>
<li>functional programming</li>
</ul>
<p>— all subjects which are dear to Tezos community, and which are the bread-and-butter of the daily work at Nomadic Labs of writing some of the most complex and safety-critical code in commercial practice.</p>
<p><span class="caps">POPL</span> took place from 17th to 22nd January 2021.
It was supposed to be in Copenhagen Denmark, but due to the pandemic it was virtualised with a semi-synchronous program catering for multiple time zones.
See <a href="https://app.clowdr.org/conference/popl2021">the full symposium events page online</a> and <a href="https://app.clowdr.org/conference/popl2021/schedule">the schedule</a>.</p>
<h2 id="our-contributions"><a class="toclink" href="#our-contributions">Our contributions</a></h2>
<p>Nomadic Labs was actively involved throughout the week:</p>
<ul>
<li>
<p>On Monday 18 Jan, <a name="dexter-pres">Arvid Jakobsson</a>, one of our experts in formal verification of smart contracts, presented <a href="https://www.youtube.com/watch?v=sFMJBTtbjTc&t=1048s">the formalisation of the Dexter <em>decentralised exchange</em> contract in the Mi-Cho-Coq framework.</a> at <a href="https://popl21.sigplan.org/room/POPL-2021-venue-cpp"><span class="caps">CPP</span>’s 2021 Lightning Talks session</a><sup id="fnref:Dexter"><a class="footnote-ref" href="#fn:Dexter">3</a></sup>.</p>
</li>
<li>
<p>On Wednesday 20 Jan, Michel Mauny, Nomadic Labs’ <span class="caps">CEO</span>, made a short presentation of our company at <a href="https://popl21.sigplan.org/details/POPL-2021-research-papers/74/Sponsor-Reception"><span class="caps">POPL</span>’s Sponsor Reception</a>, which also featured a contributed video <a href="https://www.youtube.com/watch?v=muATX5DnvnQ">presenting a general overview of Tezos by Arthur Breitman</a>.</p>
</li>
<li>
<p>On Friday 22 Jan, <a name="fcsl-pres">Germán Delbianco</a>, one of our software verification experts, together with our collaborators from <a href="https://software.imdea.org"><span class="caps">IMDEA</span> Software Institute</a> and <a href="https://ucm.es"><span class="caps">UCM</span></a>, presented an article presenting new developments on the <a href="https://app.clowdr.org/conference/popl2021/item/f35f227c-58c5-4fb3-8100-a3eebe37833c">algebraic foundations of Concurrent Separation logics.</a> at <span class="caps">POPL</span><sup id="fnref:FCSL"><a class="footnote-ref" href="#fn:FCSL">4</a></sup>.</p>
</li>
</ul>
<p>We were also pleased to find further research works backed by Tezos Foundation grants in the programs: Xuanrui Qi and Jacques Garrigue from Nagoya University, in Japan, presented advances towards <a href="https://popl21.sigplan.org/details/CoqPL-2021-papers/2/Towards-a-Coq-Specification-for-Generalized-Algebraic-Datatypes-in-OCaml">specifying OCaml <em>GADTs</em> in Coq</a>, a work funded through the <a href="http://www.math.nagoya-u.ac.jp/~garrigue/cocti/index.html">Certifiable OCaml Type Inference (<span class="caps">COCTI</span>) grant</a><sup id="fnref:cocti-sources"><a class="footnote-ref" href="#fn:cocti-sources">1</a></sup>.</p>
<!--- Tezos / NL sponsors --->
<h2 id="sponsorship"><a class="toclink" href="#sponsorship">Sponsorship</a></h2>
<ul>
<li>
<p>The Tezos Foundation was a platinum sponsor of <span class="caps">POPL</span> 2021, for the third consecutive year.</p>
</li>
<li>
<p>We at Nomadic Labs are pleased and proud to have sponsored the colocated <a href="https://popl21.sigplan.org/home/CPP-2021"><span class="caps">CPP</span> 2021 (Conference on Certified Programs and Proofs)</a>, for the second consecutive year.</p>
</li>
</ul>
<!--- POPL program -->
<h2 id="highlights"><a class="toclink" href="#highlights">Highlights</a></h2>
<p>Here are our selected highlights<sup id="fnref:fullprog"><a class="footnote-ref" href="#fn:fullprog">2</a></sup>:</p>
<h4 id="popl-2021"><a class="toclink" href="#popl-2021"><span class="caps">POPL</span> 2021</a></h4>
<p>The <a href="https://popl21.sigplan.org/track/POPL-2021-research-papers#program"><span class="caps">POPL</span> program</a> ranges from theoretical foundations of programming languages (e.g. semantics and type systems), to the development and application of formal tools and techniques for creating and verifying reliable and correct systems.</p>
<p>Selected examples include:</p>
<ul>
<li><a href="https://app.clowdr.org/conference/popl2021/item/f2ed20e6-65a3-4b1e-bc2e-4241a3452269">formalised optimisations for quantum circuits</a>;</li>
<li>programming with <a href="https://app.clowdr.org/conference/popl2021/item/6f2e6581-993f-43ae-be62-18ca6ad3fafd">and reasoning about <em>algebraic effects</em></a>;</li>
<li>formal verification of <a href="https://app.clowdr.org/conference/popl2021/item/6d106392-fbe6-4bbf-92d2-538b16a38c65">shared memory concurrency</a> and <a href="https://app.clowdr.org/conference/popl2021/item/6d0da4c1-8dc4-46d4-b813-a197f1140145"><em>probabilistic programs</em></a>; and</li>
<li><a href="https://app.clowdr.org/conference/popl2021/item/d96e88d9-98f9-4443-b19c-6a924f297ac9">how to make cryptographic primitives secure under <em>speculative executions</em></a>.</li>
</ul>
<p>Reliable blockchains and other distributed systems were the focus for two <span class="caps">POPL21</span> invited keynotes:</p>
<ul>
<li><a href="http://lpdwww.epfl.ch/rachid/index.html">Rachid Guerraoui (<span class="caps">EPFL</span>)</a> on <a href="https://popl21.sigplan.org/details/POPL-2021-research-papers/68/The-road-to-a-Universal-Internet-Machine-Demystifying-Blockchain-Protocols-">The road to a Universal Internet Machine (Demystifying Blockchain Protocols)</a> and</li>
<li><a href="https://dsf.berkeley.edu/jmh/">Joe Hellerstein (<span class="caps">UC</span> Berkeley)</a> on <a href="https://www.youtube.com/watch?v=dgOhwMmiiG0&feature=emb_title">Toward a Programmable Cloud: <span class="caps">CALM</span> Foundations and Open Challenges</a>.</li>
</ul>
<!--- CPP program -->
<h3 id="cpp-2021-conference-on-certified-programs-and-proofs"><a class="toclink" href="#cpp-2021-conference-on-certified-programs-and-proofs"><span class="caps">CPP</span> 2021 (Conference on Certified Programs and Proofs)</a></h3>
<p>As mentioned above, we sponsored <a href="https://popl21.sigplan.org/home/CPP-2021"><span class="caps">CPP</span> 2021 (Conference on Certified Programs and Proofs)</a>.
This covers mechanised verification efforts and tools, including:</p>
<ul>
<li>the formalisation of mathematics,</li>
<li>certified algorithms, and</li>
<li>the development of new proof techniques and frameworks for certified programming.</li>
</ul>
<p>A highlight of this year’s program was <a href="https://popl21.sigplan.org/details/CPP-2021/2/Invited-Talk-Underpinning-the-foundations-Sail-based-semantics-testing-and-reason">the keynote talk</a> by <a href="https://www.cl.cam.ac.uk/~pes20/">Peter Sewell (University of Cambridge)</a> accounting past, present, and future formalisation efforts towards formally specified, mechanised hardware architectures in the context of the <a href="https://www.cl.cam.ac.uk/research/security/ctsrd/cheri"><span class="caps">CHERI</span> project</a>, and their impact on the development of existing and future hardware architectures.</p>
<p>Blockchain-related topics like certified frameworks for programming correct smart contracts, or mechanised proofs of consensus algorithms, frequently appear in the program. This year, presentations included:</p>
<ul>
<li>advances in the <a href="https://popl21.sigplan.org/details/CPP-2021/25/Extracting-Smart-Contracts-Tested-and-Verified-in-Coq">ConCert framework</a> to verify, test, and extract certified smart contracts in Coq (including extraction to Liquidity and plans to target CameLIGO in the future);</li>
<li>ongoing efforts to <a href="https://www.youtube.com/watch?v=sFMJBTtbjTc&t=1760s">certify Tendermint using <span class="caps">TLA</span>+</a>;</li>
<li>and our own lightning-talk presentation on the verification of the <a href="#dexter-pres">Dexter contract</a>.</li>
</ul>
<h3 id="coqpl-2021-seventh-international-workshop-on-coq-for-programming-languages"><a class="toclink" href="#coqpl-2021-seventh-international-workshop-on-coq-for-programming-languages">CoqPL 2021 (Seventh International Workshop on Coq for Programming Languages)</a></h3>
<!--- CoqPL -->
<p>Amongst the other colocated events on offer during the week (like <a href="https://popl21.sigplan.org/home/VMCAI-2021"><span class="caps">VMCAI</span></a>, or <a href="https://popl21.sigplan.org/home/lafi-2021"><span class="caps">LAFI</span></a>), we took time to attend <a href="https://popl21.sigplan.org/home/CoqPL-2021">CoqPL 2021</a>, the Seventh International Workshop on Coq for Programming Languages.</p>
<p>The <a href="https://coq.inria.fr/">Coq proof assistant</a> is one of Nomadic Labs’ most-used verification tools, as witnessed by <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/">the Mi-Cho-Coq framework</a>, the <a href="https://clarus.github.io/coq-of-ocaml/examples/tezos/">coq-of-ocaml</a> project, and its use <a href="#dexter-pres">on this contributions to this year’s <span class="caps">CPP</span></a> and <a href="#fcsl-pres">on this contributions to this year’s <span class="caps">POPL</span></a>.</p>
<p>The Tezos Foundation supports the development of Coq and of its ecosystem, and the CoqPL workshop provides an opportunity to interact with the Coq development team, learn about the <a href="https://coq.inria.fr/news/coq-8-13-0-is-out.html">recently released features</a> and what’s coming next in the pipeline, and other recent library developments.</p>
<!--- See you next year -->
<h2 id="see-you-next-year"><a class="toclink" href="#see-you-next-year">See you next year!</a></h2>
<p><span class="caps">POPL</span> 2022 is planned for Philadelphia, <span class="caps">PA</span>. We look forward to catching up with you all again there — hopefully, in person.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:cocti-sources">
<p>You can find the <a href="https://github.com/COCTI"><span class="caps">COCTI</span> code here</a>. <a class="footnote-backref" href="#fnref:cocti-sources" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:fullprog">
<p>At time of writing, the entire program is publicly available on <a href="https://app.clowdr.org/conference/popl2021/">Clowdr</a> and should soon be uploaded to <span class="caps">SIGPLAN</span>’s <a href="https://www.youtube.com/channel/UCwG9512Wm7jSS6Iqshz4Dpg">YouTube channel</a>. <a class="footnote-backref" href="#fnref:fullprog" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:Dexter">
<p>Arvid Jakobsson, Colin González, Bruno Bernardo, and Raphaël Cauderlier. <em>Formally Verified Decentralized Exchange with Mi-Cho-Coq</em>. Contributed Lightning Talk to <a href="https://popl21.sigplan.org/room/POPL-2021-venue-cpp"><span class="caps">CPP</span> 2021</a>. Thanks also to Kristina Sojakova (<span class="caps">INRIA</span>) for her contributions to the formalisation effort. <a class="footnote-backref" href="#fnref:Dexter" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:FCSL">
<p>František Farka, Aleksandar Nanevski, Anindya Banerjee, Germán Andrés Delbianco, and Ignacio Fábregas. <em>On Algebraic Abstractions for Concurrent Separation Logics</em>. Proc. <span class="caps">ACM</span> Program. Lang. 5, <span class="caps">POPL</span>, Article 5 (January 2021), 32 pages. <a href="https://doi.org/10.1145/3434286">https://doi.org/10.1145/3434286</a> <a class="footnote-backref" href="#fnref:FCSL" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
</ol>
</div>A review of Nomadic Labs in 20202021-01-08T08:01:00+01:002021-01-08T08:01:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2021-01-08:/a-review-of-nomadic-labs-in-2020.html<p>Nomadic Labs’ activities in 2020</p><p>2020 is over and in spite of the difficulties which the year presented, we at Nomadic Labs got a lot done. So here’s who we are, and what we accomplished in 2020:</p>
<h2 id="nomadic-labs-in-a-nutshell"><a class="toclink" href="#nomadic-labs-in-a-nutshell">Nomadic Labs in a nutshell</a></h2>
<p>Nomadic Labs is an international technical company dedicated to evolving the Tezos ecosystem. Tezos is a community-driven proof-of-stake self-evolving blockchain platform that adapts and adopts new features and enables borderless global cooperation. </p>
<p>Let’s unpack that:</p>
<ul>
<li><strong>Community-driven</strong>: The Tezos community is a global community of users, researchers, and adopters (see <a href="https://tezoscommons.org">Tezos Commons</a> and <a href="https://www.tezosagora.org">Tezos Agora</a>). </li>
<li><strong>Proof-of-stake</strong>: The Tezos blockchain is based on a proof-of-stake principle, which is low-power, inclusive, and environmentally sustainable. Indeed, <a href="https://twitter.com/TBedoin/status/1110597447502491649">a Tezos node can run on a Raspberry Pi</a> (<a href="https://www.tezos.org.sg/the-berry-good-guide-to-setting-up-tezos-node-baker-using-raspberry-pi-ubuntu/">and here’s a howto!</a>).</li>
<li><strong>Self-evolving</strong>: The Tezos blockchain protocol is flexible, democratic and adaptable. We mean this in a specific technical sense — thanks to a built-in voting mechanism, the Tezos community of users can vote to <a href="#protocol-upgrade">update the protocol</a>, and <a href="#protocol-upgrade">it regularly does</a>.</li>
</ul>
<p>So Nomadic Labs contributes to a broad ecosystem dedicated to creating a resilient and global blockchain platform with associated tools. We aim to serve science, society — and the dignity and privacy of productive work in a new technological age. </p>
<p>You can find out more about us here:</p>
<ul>
<li><a href="https://youtu.be/d6iwBXKD_js">Discover Nomadic Labs in 2 min</a> (video)</li>
<li><a href="https://www.youtube.com/watch?v=dTHDAYoekAE">This short interview of our <span class="caps">CEO</span></a> by TheCoinTribune</li>
<li><a href="https://twitter.com/labosnomades?lang=en">Twitter: @LabosNomades</a> </li>
<li><a href="https://gitlab.com/nomadic-labs">Our gitlab repo</a> </li>
<li><a href="https://research-development.nomadic-labs.com/blog.html">Our technical blog</a> </li>
</ul>
<h2 id="culture-and-growth"><a class="toclink" href="#culture-and-growth">Culture and growth</a></h2>
<p>At the start of 2020 we had an office-based culture, based in <a href="https://www.openstreetmap.org/node/6260703743">beautifully-situated offices in the heart of Paris</a>. </p>
<p>Then, like everybody else, we adapted to social distancing and — as necessary — to working from home until the pandemic is over. That this adaptation was handled smoothly, was due to some genuinely hard work by administrative staff, and due to a cohesive and friendly company culture<sup id="fnref:love"><a class="footnote-ref" href="#fn:love">1</a></sup>, and we are thankful that we were able to flourish and grow throughout 2020: </p>
<ul>
<li>We started 2020 with 39 full-time employees, </li>
<li>we ended it with 59 full-time employees, and</li>
<li>we have plans for further growth in 2021.</li>
</ul>
<p>It’s also worth mentioning how well the Tezos blockchain itself withstood the 2020 stress-test. There were no hiccoughs and no stalls. Nomadic Labs — and the blockchain which is its raison d’être — have been stable in a time of crisis. The technology worked, and the community continued to grow.</p>
<h2 id="seminars-and-events"><a class="toclink" href="#seminars-and-events">Seminars and events</a></h2>
<p>In spite of the pandemic we were able to hold many meetings. Highlights include:</p>
<ul>
<li>
<p><em>We held the <a href="https://nomadic-labs.com/tezosdevday.html">Tezos Developers Day</a> (shortly before <a href="https://www.marianne.net/politique/emmanuel-macron-annonce-l-interdiction-des-deplacements-non-essentiels-des-mardi-midi">the first lockdown on 17 March</a>).</em> <br> You can enjoy the videos <a href="https://www.youtube.com/playlist?list=PLfjHt2_9f_EwthlJEiC4wRMVegPtBlL-p">here</a>. </p>
</li>
<li>
<p><em>We co-organised a <a href="http://cristal.inria.fr/~remy/Nomadic_Labs-Inria/programme.html">Journée Scientifique</a> (a one-day research workshop) with <span class="caps">INRIA</span> on 21 September 2020.</em> <br> See <a href="http://cristal.inria.fr/~remy/Nomadic_Labs-Inria/programme.html#tezos">the full programme with slides</a>. </p>
</li>
<li>
<p><em>We started a series of <a href="https://www.youtube.com/watch?v=2jd-1VxzmQ0&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7">Nomadic Labs research seminars</a>.</em> <br> Topics so far have included <a href="https://www.youtube.com/watch?v=2jd-1VxzmQ0&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=1">practical proofs using Juvix</a>, <a href="https://www.youtube.com/watch?v=SwD7fpSNnis&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=2">verifying smart contracts using Mi-Cho-Coq</a>, <a href="https://www.youtube.com/watch?v=Aha0OkKkw5A&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=3">efficient data storage on the blockchain using Plebeia</a>, and <a href="https://www.youtube.com/watch?v=mel76DFerL0&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7&index=4">adding multicore programming to OCaml</a>. More to come in 2021!</p>
</li>
<li>
<p><em>We ran <a href="https://training.nomadic-labs.com/training.html">a four-day immersive Tezos training course</a> in Feburary 2020.</em> <br> Topics included the Tezos blockchain architecture; consensus/privacy fundamentals; smart contract languages; indexers; and hands-on dapp (distributed application) building. Speakers were from Nomadic Labs, LIGOlang, Blockwatch Data Inc., <span class="caps">ECAD</span> Labs Inc., and Tezos Ukraine, presented to approximately fifty participants with backgrounds ranging from software development to the financial sector. Write to <a href="mailto:training@nomadic-labs.com">training@nomadic-labs.com</a> to learn how Nomadic Labs could help train you.</p>
</li>
</ul>
<h2 id="notable-projects-and-achievements"><a class="toclink" href="#notable-projects-and-achievements">Notable projects and achievements</a></h2>
<ul>
<li>
<p><em>The <a href="https://nomadic-labs.com/software_verification.html">Mi-Cho-Coq project</a> connects <a href="https://tezos.gitlab.io/007/michelson.html">Michelson smart contracts</a> to <a href="https://coq.inria.fr">the Coq proof assistant</a>.<sup id="fnref:pun"><a class="footnote-ref" href="#fn:pun">2</a></sup></em> <br> Writing smart contracts is safety-critical (see a <a href="https://arxiv.org/abs/1908.08605">detailed taxonomy of possible errors</a>) and Mi-Cho-Coq is an important step to help bring cutting-edge mathematics and computer science to bear on assuring safe and correct behaviour of Michelson smart contracts. </p>
</li>
<li>
<p><em>We contributed to specifying and verifying <a href="https://dexter.exchange">the Dexter exchange</a> (see also our recent <a href="https://research-development.nomadic-labs.com/dexter-decentralized-exchange-for-tezos-formal-verification-work-by-nomadic-labs.html">blog post</a>).<sup id="fnref:dexter-sources"><a class="footnote-ref" href="#fn:dexter-sources">3</a></sup></em> <br> This is a smart contract on the Tezos blockchain, for exchanging cryptocurrency tokens that are compliant with the <a href="https://assets.tqtezos.com/docs/token-contracts/fa12/1-fa12-intro/"><span class="caps">FA1</span>.2 standard</a> — this is similar to what Uniswap does on Ethereum with the <a href="https://eips.ethereum.org/EIPS/eip-20"><span class="caps">ERC20</span> standard</a>. A robust on-chain exchange mechanism is an important step towards making Tezos a globally useful cryptocurrency platform.</p>
</li>
</ul>
<h2 id="adoption"><a class="toclink" href="#adoption">Adoption</a></h2>
<h3 id="we-now-have-dedicated-adoption-and-support-teams"><a class="toclink" href="#we-now-have-dedicated-adoption-and-support-teams">We now have dedicated adoption and support teams!</a></h3>
<p>These teams are devoted to helping people and organisations to make the step into the Tezos ecosystem. You can watch a short and clear <a href="https://www.linkedin.com/posts/bsmarttv_bsmart-lachainedesaudacieux-smarttech-ugcPost-6744300980789108736-EiNG/?fbclid=IwAR1GpNfUZa64kQOA_ihBtSON5cUEVPFMD67FGXsYYq65j-zN0QIxqxP2LOY">video on stablecoins by our adoption team lead</a>. </p>
<h3 id="we-have-attracted-institutional-bakers"><a class="toclink" href="#we-have-attracted-institutional-bakers">We have attracted institutional bakers</a></h3>
<p>A “baker” on Tezos is a stakeholder (a blockchain participant) that validates operations and adds them to the Tezos blockchain (for which it is rewarded). At time of writing, Tezos has a diverse community of <a href="https://tzkt.io/bakers">more than 400 bakers worldwide</a>. </p>
<p>We are proud to announce that in 2020, our adoption and support teams provided support which helped three large institutions become bakers<sup id="fnref:fresh"><a class="footnote-ref" href="#fn:fresh">4</a></sup> on the Tezos blockchain:</p>
<ul>
<li>
<p><a href="https://en.wikipedia.org/wiki/%C3%89lectricit%C3%A9_de_France"><span class="caps">EDF</span></a> (Éléctricité de France; a multinational utility company with <a href="https://www.macrotrends.net/stocks/charts/ECIFY/electricite-de-france-sa/total-assets">340 billion <span class="caps">USD</span> in assets</a>) has become an institutional baker, via <a href="https://exaion.edf.fr/">a subsidiary Exaion</a> which it founded in January 2020. <br> <a href="https://nomadic-labs.com/download/PR-Exaion-Baker.pdf">See the press release</a>, and <a href="https://tzkt.io/tz1gcna2xxZj2eNp1LaMyAhVJ49mEFj4FH26/info">see Exaion’s official baking record</a>. </p>
</li>
<li>
<p><a href="https://www.sword-group.com/">Sword France</a> (a technological and digital transformation group founded in 2000) has become an institutional baker. As Alain Broustail (Sword Blockchain Director) said: <em>“we have been working with [Tezos] for over a year. [It] provides security guarantees, transparency and strong adaptability, which makes it very attractive”.</em> <br> <a href="https://nomadic-labs.com/download/PR-Sword-Baker.pdf">See the press release</a>. Sword is also participating in <a href="#digisign">the Tezos Digisign project</a>. </p>
</li>
<li>
<p><a href="https://smart-node.com">Smart Node</a> (a staking company) has become an institutional baker. If you hold Tez then you can delegate your stake to Smart Node, who will bake using the stake you delegated and return rewards proportional to it. <a href="https://fr.cryptonews.com/news/smartnode-devient-baker-sur-la-blockchain-tezos-8548.htm">See the announcement</a>, and <a href="https://tzkt.io/tz1NMNQYrYrfNpmUqQrMhJypf6ovnaDZhJHM/info">see Smart Node’s official baking record</a>.</p>
</li>
</ul>
<h3 id="we-helped-industrial-partners-to-launch-tezos-based-projects"><a class="toclink" href="#we-helped-industrial-partners-to-launch-tezos-based-projects">We helped industrial partners to launch Tezos-based projects</a></h3>
<ul>
<li>
<p>Société Générale (a bank with <a href="https://www.macrotrends.net/stocks/charts/SCGLY/societe-generale-group/total-assets">1.7 trillion <span class="caps">USD</span> in assets</a>) selected a pool of technology providers, including Nomadic Labs, to experiment with the use of <a href="https://www.investopedia.com/terms/c/central-bank-digital-currency-cbdc.asp">Central Bank Digital Currency (<span class="caps">CBDC</span>)</a><sup id="fnref:cbdc"><a class="footnote-ref" href="#fn:cbdc">11</a></sup> for interbank settlements. <br> Specifically, this project will explore the feasibility of financial securities being digitally settled and delivered in <span class="caps">CBDC</span>. <br> <a href="https://nomadic-labs.com/download/cp.pdf">See the press release.</a> </p>
</li>
<li>
<p><a name="digisign"></a> Sword group introduced Tezos Digisign, a <a href="https://www.sword-group.com/en/news/tezos-digisign-an-open-source-solution/">free and open source tool to sign, certify and verify the authenticity of digital documents</a>. <br> This tool is already in production with a client and is currently being integrated with several market <span class="caps">ECM</span> (Electronic Content Management) packages. The <a href="https://gitlab.com/sword-france/tezos-digisign">source code of Tezos Digisign is on Gitlab</a>. <br> <a href="https://nomadic-labs.com/download/PR-Tezos-Digisign.pdf">See the press release.</a></p>
</li>
<li>
<p>Logical Pictures is launching <a href="http://www.21contentventures.com/">21 Content Ventures</a>, an investment vehicle with a 100 million Euro maximal capacity (minimum investment 100 thousand Euro) to invest in coproducing films and <span class="caps">TV</span> series, with an emphasis on international content (e.g. from the Cannes, Toronto, and Sundance festivals).<sup id="fnref:weareone"><a class="footnote-ref" href="#fn:weareone">5</a></sup> A particularity of this investment vehicle is that it will be <em>tokenized</em> on the Tezos blockchain. This means that <ul> <li> each title will be represented on the Tezos blockchain by a security token, and <li> fundraising will take the form of a Security Token Offering (<span class="caps">STO</span>).</ul> The future portfolio of films and <span class="caps">TV</span> series will thus be digitized, which will offer more liquidity and transparency than a traditional share of funds. <br> See the press release <a href="http://www.21contentventures.com/wp-content/uploads/2020/12/21CV-PRESS-RELEASE-ENGLISH.pdf">in English</a> and <a href="http://www.21contentventures.com/wp-content/uploads/2020/12/21-CV-COMMUNIQUE-PRESSE.pdf">in French</a>.</p>
</li>
</ul>
<p><a name="protocol-upgrade"></a> </p>
<h2 id="protocol-upgrades"><a class="toclink" href="#protocol-upgrades">Protocol upgrades</a></h2>
<p>We contributed to not one but <em>two</em> Tezos protocol upgrades during 2020 (Carthage and Delphi) and we proposed a third (Edo): </p>
<p>The Tezos blockchain contains a mechanism to <a href="https://tezos.com/get-started/#self-amendment">upgrade the protocol and thus change how the blockchain works</a> by community vote, so each successful protocol upgrade is making history in the world of blockchain evolution.<sup id="fnref:unique"><a class="footnote-ref" href="#fn:unique">6</a></sup></p>
<p>Tezos protocols are traditionally named after ancient cities. Thanks to Metastate for <a href="https://core.metastate.dev/tezos-carthage-2-0-protocol-activation/">a nice timeline</a> of protocol upgrades:</p>
<ul>
<li><a href="https://tzstats.com/election/16">5 March 2020</a>. <br> Carthage (block height 851,969; cycle 208; <a href="http://tezos.gitlab.io/protocols/006_carthage.html">changelog</a>; <a href="https://research-development.nomadic-labs.com/a-new-reward-formula-for-carthage.html">significance of the upgrade</a>).</li>
<li><a href="https://tzstats.com/election/24">12 November 2020</a> <br> <a href="https://research-development.nomadic-labs.com/delphi-official-release.html">Delphi</a> (block height 1,212,417; cycle 296; <a href="https://research-development.nomadic-labs.com/delphi-changelog.html">changelog</a>; <a href="https://research-development.nomadic-labs.com/smarter-contracts-thanks-to-delphi-part-12.html">significance of the upgrade</a>).</li>
</ul>
<p>The next <a href="https://tezos.foundation/edo-proposal/">proposed protocol upgrade is Edo</a>; see <a href="https://decrypt.co/49912/tezos-will-get-zcashs-privacy-features-in-next-upgrade">an accessible explanation of what is proposed</a> (essentially: bugfixes, <a href="https://research-development.nomadic-labs.com/sapling-integration-in-tezos-tech-preview.html">Zcash Sapling integration</a>, and a ticket system for smart contracts).</p>
<p>Associated to the Edo protocol upgrade is also a new release candidate of the <a href="http://tezos.gitlab.io/releases/version-8.html">protocol environment</a>, numbered <a href="http://tezos.gitlab.io/api/odoc/_html/tezos-protocol-environment/Tezos_protocol_environment/module-type-V1/index.html">“Version 1”</a>. The <a href="http://tezos.gitlab.io/api/odoc/_html/tezos-protocol-environment/Tezos_protocol_environment/index.html"><em>protocol environment</em></a> is the set of functions that a protocol can use — a dedicated library which includes cryptographic primitives and other useful functions (packaged as <a href="http://tezos.gitlab.io/api/odoc/_html/tezos-protocol-environment/Tezos_protocol_environment/index.html">an OCaml module</a>). This is a backendish<sup id="fnref:backendish"><a class="footnote-ref" href="#fn:backendish">7</a></sup> but significant piece of work: until now all protocols have used <a href="http://tezos.gitlab.io/api/odoc/_html/tezos-protocol-environment/Tezos_protocol_environment/module-type-V0/index.html">“Version 0”</a> of the protocol environment. </p>
<h2 id="research-and-development"><a class="toclink" href="#research-and-development">Research and development</a></h2>
<ul>
<li>
<p>We started <a href="https://wp.software.imdea.org/tezos/">a research partnership with <span class="caps">IMDEA</span></a>. The emphases are on program analysis and verification, distributed consensus, resource consumption and performance, and security and privacy. <br> <a href="https://www.linkedin.com/feed/update/urn:li:activity:6623979278000287744">See the press release.</a></p>
</li>
<li>
<p>We started a research partnership with <span class="caps">INRIA</span>. This cooperation takes the form of a grants framework to support blockchain research, the results of which will be made publicly available. <br> So far, four research initiatives have been funded, employing ten researchers and two engineers. The initiatives relate to changes to OCaml and its compiler, and to the semantics of the <a href="https://www.fstar-lang.org/#tutorial">F* verification system</a>. <a href="https://inria.fr/sites/default/files/2020-03/CP%2CNomadic_labs_Inria-VF_EN_relu.pdf">See the press release.</a> <br><br> A <a href="http://cristal.inria.fr/~remy/Nomadic_Labs-Inria/programme.html">Journée Scientifique</a> with <span class="caps">INRIA</span> was held on 21 September 2020, for <a href="http://cristal.inria.fr/~remy/Nomadic_Labs-Inria/programme.html#tezos">which a full programme with slides</a> is available. </p>
</li>
<li>
<p>We advertised <a href="https://nomadic-labs.com/internships.html">ten internships</a>, which are open at the time of writing this summary. Feel free to send in an application! </p>
</li>
<li>
<p>We spawned <a href="https://tezos.gitlab.io/introduction/test_networks.html">several test networks in 2020</a>. A test network (<em>testnet</em>, for short) is a ‘mock’ blockchain where the rules are tweaked to allow experimentation e.g. before pushing a protocol upgrade. The test networks were:</p>
<ul>
<li><a href="https://tezos.gitlab.io/introduction/test_networks.html#dalphanet">Dalphanet</a>. <br> This was spawned during summer 2020 and is now closed. It was used to test <a href="https://research-development.nomadic-labs.com/sapling-integration-in-tezos-tech-preview.html">Sapling integration</a> and <a href="https://medium.com/metastatedev/stateful-baking-accounts-d483a63a173c">baking accounts</a>.</li>
<li><a href="https://tezos.gitlab.io/introduction/test_networks.html#delphinet">Delphinet</a>. <br> This was used to test the <a href="https://decrypt.co/48019/tezos-delphi-upgrade-slashes-gas-costs-by-75">Delphi protocol upgrade</a> (which was voted for and accepted on 12 November 2020). See also <a href="https://research-development.nomadic-labs.com/smarter-contracts-thanks-to-delphi-part-12.html">a full account of the significance of this upgrade</a>.</li>
<li><a href="https://tezos.gitlab.io/introduction/test_networks.html#edonet">Edonet</a>. <br> This is a current test network being used to test the <a href="https://research-development.nomadic-labs.com/announcing-the-edo-release.html">Edo protocol</a>.</li>
</ul>
</li>
<li>
<p>We commonly open-source the tools we use to develop our software. For example, we <a href="https://tezos.gitlab.io/user/mockup.html">released <code>mockup mode</code></a>, a tool in the <a href="https://tezos.gitlab.io/introduction/howtoget.html">tezos client</a> for experimenting with (parts of) the essential <span class="caps">API</span> of a Tezos node without having to run a local blockchain or maintain a consensus algorithm. This has been especially useful to our developers working on smart contracts. We <a href="https://research-development.nomadic-labs.com/introducing-mockup-mode-for-tezos-client.html">extensively documented</a> this new tool’s usage. </p>
</li>
<li>
<p>We considerably extended our analysis of Emmy<sup>+</sup>. <br> Emmy<sup>+</sup> is the current Tezos consensus algorithm, so it is important that we understand its behaviour both in theory and empirically. We extended the <a href="https://research-development.nomadic-labs.com/analysis-of-emmy.html">initial analysis from 2019</a> and presented the results as follows:</p>
<ul>
<li>A <a href="https://research-development.nomadic-labs.com/on-defending-against-malicious-reorgs-in-tezos-proof-of-stake.html">study of malicious reorgs</a>.</li>
<li>A <a href="https://research-development.nomadic-labs.com/emmy-in-the-partial-synchrony-model.html">study of partial synchrony</a>.</li>
<li>A <a href="https://research-development.nomadic-labs.com/the-case-of-mixed-forks-in-emmy.html">study of mixed forks</a>.</li>
</ul>
</li>
<li>
<p>We proposed <a href="https://arxiv.org/abs/2001.11965">Tenderbake</a>: an adaptation of <a href="https://arxiv.org/abs/1807.04938">the Tendermint algorithm</a> for Tezos, in collaboration with <a href="http://www-list.cea.fr/en/"><span class="caps">CEA</span>-List</a>. <br> We have a Tenderbake prototype and are integrating it in the economic protocol as a next-gen upgrade from Emmy<sup>+</sup>.<sup id="fnref:deterministic-finality"><a class="footnote-ref" href="#fn:deterministic-finality">8</a></sup> We expect to release a Tenderbake testnet in 2021. </p>
</li>
<li>
<p>If you want to design your own Tezos protocol, these two blog posts might help: <a href="https://research-development.nomadic-labs.com/how-to-write-a-tezos-protocol.html">part 1</a> and <a href="https://research-development.nomadic-labs.com/how-to-write-a-tezos-protocol-part-2.html">part 2</a>.</p>
</li>
<li>
<p>We’re developers, and developers test their programs. But how do you know your coverage is good — that your tests cover the important cases? Well, here are two posts on our use of testing tools: <ul> <li> <a href="https://research-development.nomadic-labs.com/catching-sneaky-regressions-with-pytest-regtest.html"><code>pytest</code></a> and <li> <a href="https://research-development.nomadic-labs.com/measuring-test-coverage.html"><code>bisect_ppx</code></a>. </ul> There’s <a href="https://nomadic-labs.com/pages/quick-but-correct-invariant-powered-property-testing.html">an open internship</a> if you’re interested, and you can get a feel for how this work gets done in practice by looking at <a href="https://gitlab.com/nomadic-labs/tezos/-/issues/63">this merge request for applying the <span class="caps">TZT</span> framework to test Michelson expressions</a>.</p>
</li>
<li>
<p><a href="https://clarus.github.io/coq-of-ocaml/examples/tezos/">We formalized</a> most of the <a href="https://gitlab.com/tezos/tezos/-/tree/master/src/proto_alpha/lib_protocol">Tezos economic protocol</a> in Coq, using the <a href="https://clarus.github.io/coq-of-ocaml/">coq-of-ocaml tool</a>, itself developed at Nomadic Labs. This is part of a larger project aiming to formally verify — meaning, to formally represent, and then verify using a computer correct behaviour of — the Tezos implementation itself. </p>
</li>
<li>
<p>We presented <a href="https://albert-lang.io">Albert</a>, an intermediate smart contract programming language compiled to Michelson.<sup id="fnref:albert"><a class="footnote-ref" href="#fn:albert">10</a></sup> Albert is an imperative language with variables and records — abstracting away from Michelson’s stack-based paradigm — intended as a compilation target for higher-level smart contract programming languages. See <a href="https://arxiv.org/abs/2001.02630">the paper</a> in <a href="https://fc20.ifca.ai/wtsc/"><span class="caps">WTSC</span> 2020</a>. Albert’s compiler is written in <a href="https://coq.inria.fr">Coq</a>, which means that we can certify it: currently, the backend optimizer <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/merge_requests/53">is certified</a>.<sup id="fnref:certified"><a class="footnote-ref" href="#fn:certified">9</a></sup></p>
</li>
</ul>
<h2 id="further-reading-listening-viewing"><a class="toclink" href="#further-reading-listening-viewing">Further reading / listening / viewing</a></h2>
<ul>
<li><a href="https://tezos.foundation/tezos-2020-year-in-review/">Tezos year in review</a>: a review of the year 2020 from the point of view of the Tezos foundation.</li>
<li><a href="https://research-development.nomadic-labs.com/blog.html">Nomadic Labs blog</a>: an acccount of all the wonderful things we thought to blog about, going all the way back to our <a href="https://research-development.nomadic-labs.com/hello-world.html">first wonderful blog post</a>. </li>
<li><a href="https://research-development.nomadic-labs.com/blog.html?category=meanwhile">Meanwhile at Nomadic Labs</a>: a (roughly) monthly update on our activities.</li>
<li><a href="https://tquorum.com/stream">The TQuorum podcast</a> (also available on <a href="https://www.amazon.com/TQuorum-Sessions/dp/B08K58DY16">audible</a>): a weekly podcast about all things Tezos. </li>
<li><a href="https://www.youtube.com/c/TQTezos/videos">TQTezos</a> is full of videos on anything and everything related to Tezos. </li>
<li><a href="https://forum.tezosagora.org">Tezos Agora</a>: a place to discuss and learn about the long-term direction of the Tezos project.</li>
<li><a href="https://www.youtube.com/watch?v=2jd-1VxzmQ0&list=PLfjHt2_9f_EwJh1Yr3zgRZnqrc-koohC7">Nomadic Labs research seminars</a> and <a href="https://nomadic-labs.com/software_verification.html">more information on research</a>.</li>
<li><a href="https://www.youtube.com/channel/UCp9imUAFtU5kRyFKHN2iCDA">Nomadic Labs YouTube channel</a>.</li>
</ul>
<p>Nomadic Labs moves into 2021 from a position of consolidation, growth, and strength. We look forward to the new year’s technical challenges and wish our readers, users, and community all the best. Happy new year!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:love">
<p>Not a platitude. Company culture is real, as are the returns in productivity when management gets this right. <a class="footnote-backref" href="#fnref:love" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:pun">
<p>The name is a pun on a <a href="https://www.carambarco.com/marques/michoko/">popular French sweet</a>. <a class="footnote-backref" href="#fnref:pun" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:dexter-sources">
<p>We used <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/">Mi-Cho-Coq</a>, and you can find <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/merge_requests/71">the full sources here on Gilab</a>. <a class="footnote-backref" href="#fnref:dexter-sources" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:fresh">
<p>… making them <em>fresh</em> bakers (one bun baguettes another). <a class="footnote-backref" href="#fnref:fresh" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:weareone">
<p>This is an ongoing fundraising effort and the amount raised will be up to 100 million Euro. Fun fact: these and other festivals were <a href="https://www.youtube.com/weareone">streamed as the <span class="caps">WE</span> <span class="caps">ARE</span> <span class="caps">ONE</span> global film festival</a> in 2020. <a class="footnote-backref" href="#fnref:weareone" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:unique">
<p>See <a href="https://tezos.com/static/position_paper-841a0a56b573afb28da16f6650152fb4.pdf">the 2014 Tezos whitepaper</a>. The technical term for this is <em>governance</em>. Tezos’ governance features are a <span class="caps">USP</span> of the Tezos blockchain and they are <em>almost</em> unique in the blockchain space: see also <a href="https://wiki.polkadot.network/docs/en/learn-governance">Polkadot</a> governance, initiated circa 2017. <a href="https://docs.decred.org/governance/overview/">Decred</a>, <a href="https://whitepaperdatabase.com/decred-dcr-whitepaper/">also from 2017</a>, also has a governance system but it is part on-chain and part off-chain: voting itself is on-chain; but after the vote any upgrades have to be done manually. <a class="footnote-backref" href="#fnref:unique" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:backendish">
<p><em>backendish</em>: of or pertaining to the back-end of a system or process. Coined in this blog post. Not to be confused with the <a href="https://en.wiktionary.org/wiki/backendish">rare Yorkshire word for “autumnal”</a>. <a class="footnote-backref" href="#fnref:backendish" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn:deterministic-finality">
<p>The key <span class="caps">USP</span> of Tenderbake vs Emmy<sup>+</sup> is that Tenderbake has <em>deterministic finality</em> whereas Emmy<sup>+</sup> has <em>probabilistic finality</em>. To simplify: deterministic finality means you will reach consensus, whereas probabilistic finality means you will <em>probably</em> reach consensus, eventually. The problem here is not with “probably”. The problem is with “eventually”: if the network is subject to delays then as the maths works out, “eventually” could become a very long time (though, what attacker would be so impolite as to DoS a system’s communications network?). <a class="footnote-backref" href="#fnref:deterministic-finality" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
<li id="fn:certified">
<p>The backend optimizer of the Albert compiler is a Coq function that transforms Michelson expressions into Michelson expressions. <em>Certifying the backend optimizer</em> means we proved mathematically in Coq that it preserves typing and semantics; that is, that the transformation’s output program is semantically equivalent to its input program. For a concrete example of what Nomadic Labs produces, you can find that last sentence written up as code and proved in Coq by one of our engineers, <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/blob/dev/src/michocoq/typed_optimizer.v#L2120">here</a>. <a class="footnote-backref" href="#fnref:certified" title="Jump back to footnote 9 in the text">↩</a></p>
</li>
<li id="fn:albert">
<p>Albert is called “Albert” after <a href="https://en.wikipedia.org/wiki/Albert_A._Michelson">Albert Michelson</a>, a Nobel prizewinning physicist born in 1851 who is the “Michelson” in the <a href="https://en.wikipedia.org/wiki/Michelson–Morley_experiment">Michelson-Morley experiment</a>. With extraordinary foresight, his family named themselves after the foundational smart contracts language of the Tezos blockchain. <a class="footnote-backref" href="#fnref:albert" title="Jump back to footnote 10 in the text">↩</a></p>
</li>
<li id="fn:cbdc">
<p>A Central Bank Digital Currency uses digital tokens on a blockchain to represent a country’s fiat currency. This allows a blockchain and smart contracts system to serve as a platform for transactions in fiat currency. <a class="footnote-backref" href="#fnref:cbdc" title="Jump back to footnote 11 in the text">↩</a></p>
</li>
</ol>
</div>Introducing mockup mode for tezos-client2020-12-21T15:00:00+01:002020-12-21T15:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2020-12-21:/introducing-mockup-mode-for-tezos-client.html<p>Presenting tezos-client’s new mockup mode feature</p><p>We are pleased to announce that <a href="https://tezos.gitlab.io/introduction/howtoget.html">the <code>tezos-client</code> binary</a>
has a new feature aimed at contract and tool developers alike: the <strong>mockup mode</strong>.</p>
<p>Mockup mode allows easy prototyping of Tezos applications and smart contracts <em>locally</em>.
<a name="local"></a> By <strong>local</strong> we mean: </p>
<ul>
<li>The relevant data files sit in a directory on your computer’s local filesystem. </li>
<li>These files are a lightweight emulation of the internal state of a Tezos single-node network. </li>
<li>Thus, networking communications infrastructure that a node would be wrapped in, has been stripped away.</li>
<li>Likewise, the consensus mechanisms that would be needed for a live blockchain with a network and multiple nodes, have been stripped away. </li>
<li>The state is directly accessible and modifiable, since it’s just files on your computer. </li>
<li>You don’t need (complex) setups of node-client interactions. There is no infrastructure aside from your own filesystem with a bundle of state files in a directory. It’s easy to inspect these files, modify them, and play with different configurations and protocols.</li>
</ul>
<p>If a single <a href="#differences-from-sandboxed-mode">sandboxed node</a> were an apricot then mockup mode would be the apricot kernel, and we could write:</p>
<div class="highlight"><pre><span></span><code> mockup mode = kernel_of(one sandboxed Tezos node)
</code></pre></div>
<p>Our motivation in building mockup mode was to give our developers, who are building and testing <a href="https://tezos.gitlab.io/007/michelson.html">Tezos smart contracts</a>, an easy local environment offering a fast development cycle which needs only lightweight local state files, and which does not require a running blockchain. Now, we’d like to share the joy of this new tool with you. </p>
<p>The features described below are available <a href="https://gitlab.com/tezos/tezos/-/tree/master">on
the master branch</a>, and will be
included in <a href="https://gitlab.com/tezos/tezos/-/tree/v8-release">version <code>8.0</code></a>.<sup id="fnref:versions"><a class="footnote-ref" href="#fn:versions">1</a></sup></p>
<p>This post is a practical guide to mockup mode’s features (see also the <a href="https://tezos.gitlab.io/user/mockup.html">documentation</a>). Be prepared for lots of command-line snippets, which you are welcome to run for yourself!</p>
<ul>
<li><a href="#overview">Overview</a><ul>
<li><a href="#the-basic-command---tezos-client-">The basic command: <code>tezos-client</code></a></li>
<li><a href="#three-modes-of-operation">Three modes of operation</a></li>
<li><a href="#capabilities-of-mockup-mode">Capabilities of mockup mode</a></li>
</ul>
</li>
<li><a href="#stateless_mockup">Run a mockup client in stateless mode</a><ul>
<li><a href="#typecheck-and-evaluate-scripts">Typecheck and evaluate scripts</a></li>
<li><a href="#query-available-mockup-protocols">Query available mockup protocols</a></li>
</ul>
</li>
<li><a href="#stateful_mockup">Run a mockup client with state</a><ul>
<li><a href="#mockup-client">A useful command alias: <code>mockup-client</code></a></li>
<li><a href="#our-first-state">Our first state</a></li>
<li><a href="#something-more-advanced--interacting-with-contracts">Something more advanced: interacting with contracts</a></li>
</ul>
</li>
<li><a href="#tune-mockup-parameters">Tune mockup parameters</a><ul>
<li><a href="#context-state">Context state</a></li>
</ul>
</li>
<li><a href="#asynchronous-mockup">Running a mockup client with asynchronous state</a><ul>
<li><a href="#use">Use</a></li>
<li><a href="#differences-from-sandboxed-mode">Differences from sandboxed mode</a></li>
</ul>
</li>
<li><a href="#conclusions">Conclusions</a></li>
</ul>
<h2 id="overview"><a class="toclink" href="#overview">Overview</a></h2>
<h3 id="the-basic-command-tezos-client"><a class="toclink" href="#the-basic-command-tezos-client">The basic command: <code>tezos-client</code></a></h3>
<p><code>tezos-client</code> is the main tool for advanced user interaction with the Tezos blockchain. </p>
<p><code>tezos-client</code> can prepare transactions; evaluate, typecheck and originate
contracts; and encode/decode data when interacting with nodes. It also acts as
a wallet, allowing to sign arbitrary data — including, of course, transactions.</p>
<p>The mockup mode of <code>tezos-client</code> supports these operations (with slight limitations<sup id="fnref:limit"><a class="footnote-ref" href="#fn:limit">2</a></sup>), with the convenience that it does not need to be connected to a live Tezos node.
All operations are <a href="#local">local, in the sense above</a>. </p>
<p><code>tezos-client</code> in mockup mode does two things to compensate for not communicating with the live network:</p>
<ol>
<li>It allows the user to specify — or if none are specified, it invents — dummy values for required initialisation parameters which would usually be gathered from a live node. Examples include: the head of the chain; or the client’s network identifier. </li>
<li>It simulates activation of the protocol, and runs local implementations of the <strong><span class="caps">RPC</span></strong>s (<strong>Remote Procedure Calls</strong>).</li>
</ol>
<h3 id="three-modes-of-operation"><a class="toclink" href="#three-modes-of-operation">Three modes of operation</a></h3>
<p>Mockup mode can run in three ways:</p>
<ol>
<li><a href="#stateless_mockup">Stateless mode</a>. <br> In this mode, <code>tezos-client</code> operates on its inputs and returns a value. Nothing is written to disk, and no state is preserved between calls to the client. This is the default. </li>
<li><a href="#stateful_mockup">Stateful mode</a>. <br> In this mode, <code>tezos-client</code> creates or manipulates a state on disk. The switch for this is <code>--base-dir <directory_name></code>; <a href="#alias_definition"">example here</a>.</li>
<li><a href="#asynchronous_mockup">Stateful asynchronous mode</a>. <br> This mode adds baking. The command-line switch for this is <code>--base-dir <directory_name> --asynchronous</code>; <a href="#asynchronous_mockup_example">example here</a>. </li>
</ol>
<p><a name="mockup-capabilities"></a></p>
<h3 id="capabilities-of-mockup-mode"><a class="toclink" href="#capabilities-of-mockup-mode">Capabilities of mockup mode</a></h3>
<p>The current implementation of mockup mode can:</p>
<ul>
<li><em>Typecheck, serialize, sign and evaluate a contract.</em>
<br> These features work in stateless mode.</li>
<li><em>Perform transactions, originations, and contract calls</em> —
mimicking <a href="https://tezos.gitlab.io/user/sandbox.html">sandboxed mode</a> but without a node.
<br> These features require a state.</li>
<li><em>Register delegates and bake blocks.</em>
<br> These features require an asynchronous state.</li>
</ul>
<p>In practice we find it simplest to just use state and remember to delete it between sessions — but your needs may vary and the tool will accomodate them. </p>
<p>We will now consider the capabilities in more detail.</p>
<h2 id="run-a-mockup-client-in-stateless-mode"><a class="toclink" href="#run-a-mockup-client-in-stateless-mode">Run a mockup client in stateless mode <a name="stateless_mockup"></a></a></h2>
<h3 id="typecheck-and-evaluate-scripts"><a class="toclink" href="#typecheck-and-evaluate-scripts">Typecheck and evaluate scripts</a></h3>
<p>The mockup mode can typecheck and evaluate scripts. Let’s try this on a script
<code>hardlimit.tz</code>, which you can download from
<a href="https://gitlab.com/tezos/tezos/-/blob/master/tests_python/contracts_alpha/mini_scenarios/hardlimit.tz">Tezos master branch</a>
or create locally as follows:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>><span class="w"> </span>hardlimit.tz<span class="w"> </span><span class="s"><<EOF</span>
<span class="s">parameter unit ;</span>
<span class="s">storage int ;</span>
<span class="s">code { # This contract stops accepting transactions after N incoming transactions</span>
<span class="s"> CDR ; DUP ; PUSH int 0 ; CMPLT; IF {PUSH int -1 ; ADD} {FAIL};</span>
<span class="s"> NIL operation ; PAIR} ;</span>
<span class="s">EOF</span>
</code></pre></div>
<ul>
<li>
Typechecking a script:
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-client<span class="w"> </span>--protocol<span class="w"> </span>ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--mode<span class="w"> </span>mockup<span class="w"> </span>typecheck<span class="w"> </span>script<span class="w"> </span>hardlimit.tz
</code></pre></div>
<div class="highlight"><pre><span></span><code>Well typed
Gas remaining: 1039988.27 units remaining
</code></pre></div>
<li>
Evaluating a script:
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-client<span class="w"> </span>--protocol<span class="w"> </span>ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--mode<span class="w"> </span>mockup<span class="w"> </span>run<span class="w"> </span>script<span class="w"> </span>hardlimit.tz<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>on<span class="w"> </span>storage<span class="w"> </span><span class="s1">'2'</span><span class="w"> </span>and<span class="w"> </span>input<span class="w"> </span><span class="s1">'Unit'</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code> storage
1
emitted operations
big_map diff
</code></pre></div>
</ul>
<p>Without the <code>--protocol</code> option, the mockup mode will choose a protocol for you.<sup id="fnref:technical"><a class="footnote-ref" href="#fn:technical">3</a></sup></p>
<h3 id="query-available-mockup-protocols"><a class="toclink" href="#query-available-mockup-protocols">Query available mockup protocols</a></h3>
<p>We can query the list of the Tezos protocols that mockup mode supports: </p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-client<span class="w"> </span>list<span class="w"> </span>mockup<span class="w"> </span>protocols
</code></pre></div>
<p>As this article went to print (so to speak), this command returns three protocol identifiers (ignoring any <code>Warnings</code>):</p>
<div class="highlight"><pre><span></span><code>ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK
PsDELPH1Kxsxt8f9eWbxQeRxkjfbxoqM52jvs5Y5fBxWWh4ifpo
PsCARTHAGazKbHtnKfLzQg3kms52kSRpgnDY982a9oYsSXRLQEb
</code></pre></div>
<p><a name="naming-protocols"></a>
<em>A word on the protocol for naming protocols:</em> The Tezos protocol IDs above are based on hashes, but the start of each <span class="caps">ID</span> hints at the release name of the corresponding protocol. The three items above correspond to protocols called <code>alpha</code> (a development version of the Tezos protocol), <code>Delphi</code>, and <code>Carthage</code>.</p>
<p>Getting these IDs matters because a Tezos blockchain requires a protocol, thus in particular setting up a <em>mockup</em> state requires us to choose a protocol.
The list above tells us what’s available.</p>
<h2 id="run-a-mockup-client-with-state"><a class="toclink" href="#run-a-mockup-client-with-state">Run a mockup client with state <a name="stateful_mockup"></a></a></h2>
<p>Giving the mockup client some state allows access more of the <a href="#mockup-capabilities">available functionalities</a>.
In particular, given a state we can operate on it, including:</p>
<ul>
<li>transferring Tez cryptocurrency tokens (ꜩ), </li>
<li>originating (deploying) contracts, </li>
<li>importing keys, and </li>
<li>querying balances or (more generally) making <span class="caps">RPC</span> queries on the chain’s current state. </li>
</ul>
<h3 id="a-useful-command-alias-mockup-client"><a class="toclink" href="#a-useful-command-alias-mockup-client">A useful command alias: <code>mockup-client</code> <a name="mockup-client"></a></a></h3>
<p>A shell alias will let us call <code>tezos-client</code> with <code>--mode mockup</code> and <code>--base-dir /tmp/mockup</code>, and so save us keystrokes later: </p>
<p><a name="alias_definition"></a></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span><span class="nb">alias</span><span class="w"> </span>mockup-client<span class="o">=</span><span class="s1">'tezos-client --mode mockup --base-dir /tmp/mockup'</span>
</code></pre></div>
<h3 id="our-first-state"><a class="toclink" href="#our-first-state">Our first state</a></h3>
<p>Time to make a mockup session with some state!</p>
<h4 id="making-the-state"><a class="toclink" href="#making-the-state">Making the state</a></h4>
<p><a name="making_the_state_command"></a></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>--protocol<span class="w"> </span>ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK<span class="w"> </span>create<span class="w"> </span>mockup
</code></pre></div>
<div class="highlight"><pre><span></span><code>Created mockup client base dir in /tmp/mockup
Tezos address added: tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx
Tezos address added: tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN
Tezos address added: tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU
Tezos address added: tz1b7tUupMgCNw2cCLpKTkSD1NZzB5TkP2sv
Tezos address added: tz1ddb9NMYHZi5UzPdzTZMYQQZoMub195zgv
</code></pre></div>
<p>Note that:</p>
<ul>
<li>The state is stored in <code>/tmp/mockup</code> because of the <a href="#mockup-client"><code>--base-dir</code> option in <code>mockup-client</code></a>.</li>
<li>The switch <code>--protocol ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK</code> <a href="#naming-protocols">means that</a> this mockup session will use protocol <code>alpha</code> for all subsequent commands on this state (see next point). </li>
<li>Mockup mode does <em>not</em> support protocol updates<sup id="fnref:updates"><a class="footnote-ref" href="#fn:updates">4</a></sup>, so if we want a new protocol we need to start from a new state. Thus, for this session we are stuck with our initial choice of <code>alpha</code>. In the near future, mockup mode will support <a href="https://gitlab.com/tezos/tezos/-/merge_requests/2315">protocol migrations</a> to facilitate protocol switching.</li>
</ul>
<p>The output above confirms that: </p>
<ul>
<li>A mockup state data directory <code>/tmp/mockup</code> has been created. Data is <ul>
<li>in <code>/tmp/mockup</code> for non-mockup-specific elements (like accounts), and </li>
<li>in <code>/tmp/mockup/mockup</code> for mockup-specific data (like mempool, trashpool, and context) — see <a href="#asynchronous_mockup">asynchronous state</a> for details). </li>
</ul>
</li>
<li>Five accounts have been added, and their addresses are listed. </li>
</ul>
<p>The five accounts are called <code>bootstrap1</code> to <code>bootstrap5</code> (see <a href="#bootstraps">command below</a>). The reader familiar with Tezos’ sandboxed client may recognize them as <a href="https://tezos.gitlab.io/user/sandbox.html#use-the-sandboxed-client">the preconfigured <code>bootstrap1</code> to <code>bootstrap5</code> accounts which it creates</a>. <!--(See also <a href="#differences-from-sandboxed-mode">the comparison below</a> of mockup and sandboxing.)--></p>
<h4 id="list-known-addresses"><a class="toclink" href="#list-known-addresses">List known addresses</a></h4>
<p>We can now use standard commands, e.g., to list known addresses.</p>
<p><a name="bootstraps"></a></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>list<span class="w"> </span>known<span class="w"> </span>addresses
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="n">bootstrap5</span><span class="o">:</span><span class="w"> </span><span class="n">tz1ddb9NMYHZi5UzPdzTZMYQQZoMub195zgv</span><span class="w"> </span><span class="o">(</span><span class="n">unencrypted</span><span class="w"> </span><span class="n">sk</span><span class="w"> </span><span class="n">known</span><span class="o">)</span>
<span class="n">bootstrap4</span><span class="o">:</span><span class="w"> </span><span class="n">tz1b7tUupMgCNw2cCLpKTkSD1NZzB5TkP2sv</span><span class="w"> </span><span class="o">(</span><span class="n">unencrypted</span><span class="w"> </span><span class="n">sk</span><span class="w"> </span><span class="n">known</span><span class="o">)</span>
<span class="n">bootstrap3</span><span class="o">:</span><span class="w"> </span><span class="n">tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU</span><span class="w"> </span><span class="o">(</span><span class="n">unencrypted</span><span class="w"> </span><span class="n">sk</span><span class="w"> </span><span class="n">known</span><span class="o">)</span>
<span class="n">bootstrap2</span><span class="o">:</span><span class="w"> </span><span class="n">tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN</span><span class="w"> </span><span class="o">(</span><span class="n">unencrypted</span><span class="w"> </span><span class="n">sk</span><span class="w"> </span><span class="n">known</span><span class="o">)</span>
<span class="n">bootstrap1</span><span class="o">:</span><span class="w"> </span><span class="n">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span><span class="w"> </span><span class="o">(</span><span class="n">unencrypted</span><span class="w"> </span><span class="n">sk</span><span class="w"> </span><span class="n">known</span><span class="o">)</span>
</code></pre></div>
<h4 id="transfer-tokens"><a class="toclink" href="#transfer-tokens">Transfer tokens</a></h4>
<p>We can execute the canonical example of a transfer from one address to
another.<sup id="fnref:immediate"><a class="footnote-ref" href="#fn:immediate">5</a></sup></p>
<p><a name="stateful-mockup-first-transfer"></a></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>transfer<span class="w"> </span><span class="m">100</span><span class="w"> </span>from<span class="w"> </span>bootstrap1<span class="w"> </span>to<span class="w"> </span>bootstrap2
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="nv">Node</span><span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="nv">bootstrapped</span>.
<span class="nv">Estimated</span><span class="w"> </span><span class="nv">gas</span>:<span class="w"> </span><span class="mi">1427</span><span class="w"> </span><span class="nv">units</span><span class="w"> </span><span class="ss">(</span><span class="nv">will</span><span class="w"> </span><span class="nv">add</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">safety</span><span class="ss">)</span>
<span class="nv">Estimated</span><span class="w"> </span><span class="nv">storage</span>:<span class="w"> </span><span class="nv">no</span><span class="w"> </span><span class="nv">bytes</span><span class="w"> </span><span class="nv">added</span>
<span class="nv">Operation</span><span class="w"> </span><span class="nv">successfully</span><span class="w"> </span><span class="nv">injected</span><span class="w"> </span><span class="nv">in</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">node</span>.
<span class="nv">Operation</span><span class="w"> </span><span class="nv">hash</span><span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="s1">'ooVjVsPgUuy4grpDBbKr5QPc667JCQ6nbMeqeTjqiRzXiCiy5e9'</span>
<span class="nv">NOT</span><span class="w"> </span><span class="nv">waiting</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">operation</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">be</span><span class="w"> </span><span class="nv">included</span>.
<span class="nv">Use</span><span class="w"> </span><span class="nv">command</span>
<span class="w"> </span><span class="nv">tezos</span><span class="o">-</span><span class="nv">client</span><span class="w"> </span><span class="k">wait</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">ooVjVsPgUuy4grpDBbKr5QPc667JCQ6nbMeqeTjqiRzXiCiy5e9</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">be</span><span class="w"> </span><span class="nv">included</span><span class="w"> </span><span class="o">--</span><span class="nv">confirmations</span><span class="w"> </span><span class="mi">30</span><span class="w"> </span><span class="o">--</span><span class="nv">branch</span><span class="w"> </span><span class="nv">BLockGenesisGenesisGenesisGenesisGenesisCCCCCeZiLHU</span>
<span class="nv">and</span><span class="o">/</span><span class="nv">or</span><span class="w"> </span><span class="nv">an</span><span class="w"> </span><span class="nv">external</span><span class="w"> </span><span class="nv">block</span><span class="w"> </span><span class="nv">explorer</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">make</span><span class="w"> </span><span class="nv">sure</span><span class="w"> </span><span class="nv">that</span><span class="w"> </span><span class="nv">it</span><span class="w"> </span><span class="nv">has</span><span class="w"> </span><span class="nv">been</span><span class="w"> </span><span class="nv">included</span>.
<span class="nv">This</span><span class="w"> </span><span class="nv">sequence</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">operations</span><span class="w"> </span><span class="nv">was</span><span class="w"> </span><span class="nv">run</span>:
<span class="w"> </span><span class="nv">Manager</span><span class="w"> </span><span class="nv">signed</span><span class="w"> </span><span class="nv">operations</span>:
<span class="w"> </span><span class="nv">From</span>:<span class="w"> </span><span class="nv">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span>
<span class="w"> </span><span class="nv">Fee</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">baker</span>:<span class="w"> </span>ꜩ<span class="mi">0</span>.<span class="mi">000404</span>
<span class="w"> </span><span class="nv">Expected</span><span class="w"> </span><span class="nv">counter</span>:<span class="w"> </span><span class="mi">1</span>
<span class="w"> </span><span class="nv">Gas</span><span class="w"> </span><span class="nv">limit</span>:<span class="w"> </span><span class="mi">1527</span>
<span class="w"> </span><span class="nv">Storage</span><span class="w"> </span><span class="nv">limit</span>:<span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nv">bytes</span>
<span class="w"> </span><span class="nv">Balance</span><span class="w"> </span><span class="nv">updates</span>:
<span class="w"> </span><span class="nv">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span><span class="w"> </span>................<span class="w"> </span><span class="o">-</span>ꜩ<span class="mi">0</span>.<span class="mi">000404</span>
<span class="w"> </span><span class="nv">fees</span><span class="ss">(</span><span class="nv">the</span><span class="w"> </span><span class="nv">baker</span><span class="w"> </span><span class="nv">who</span><span class="w"> </span><span class="nv">will</span><span class="w"> </span><span class="k">include</span><span class="w"> </span><span class="nv">this</span><span class="w"> </span><span class="nv">operation</span>,<span class="mi">0</span><span class="ss">)</span><span class="w"> </span>...<span class="w"> </span><span class="o">+</span>ꜩ<span class="mi">0</span>.<span class="mi">000404</span>
<span class="w"> </span><span class="nv">Transaction</span>:
<span class="w"> </span><span class="nv">Amount</span>:<span class="w"> </span>ꜩ<span class="mi">100</span>
<span class="w"> </span><span class="nv">From</span>:<span class="w"> </span><span class="nv">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span>
<span class="w"> </span><span class="nv">To</span>:<span class="w"> </span><span class="nv">tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN</span>
<span class="w"> </span><span class="nv">This</span><span class="w"> </span><span class="nv">transaction</span><span class="w"> </span><span class="nv">was</span><span class="w"> </span><span class="nv">successfully</span><span class="w"> </span><span class="nv">applied</span>
<span class="w"> </span><span class="nv">Consumed</span><span class="w"> </span><span class="nv">gas</span>:<span class="w"> </span><span class="mi">1427</span>
<span class="w"> </span><span class="nv">Balance</span><span class="w"> </span><span class="nv">updates</span>:
<span class="w"> </span><span class="nv">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span><span class="w"> </span>...<span class="w"> </span><span class="o">-</span>ꜩ<span class="mi">100</span>
<span class="w"> </span><span class="nv">tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN</span><span class="w"> </span>...<span class="w"> </span><span class="o">+</span>ꜩ<span class="mi">100</span>
</code></pre></div>
<p>Let’s check that the transfer has been registered:</p>
<ol>
<li>
First, we check that the sender <code>bootstrap1</code> has indeed paid the transfer amount, plus fees:
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>get<span class="w"> </span>balance<span class="w"> </span><span class="k">for</span><span class="w"> </span>bootstrap1
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="mf">3999899.999596</span><span class="w"> </span><span class="n">ꜩ</span>
</code></pre></div>
<li>
Second, we check that the receiver <code>bootstrap2</code> indeed has an extra 100 ꜩ in its balance:
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>get<span class="w"> </span>balance<span class="w"> </span><span class="k">for</span><span class="w"> </span>bootstrap2
</code></pre></div>
<div class="highlight"><pre><span></span><code> 4000100 ꜩ
</code></pre></div>
</ol>
<h3 id="something-more-advanced-interacting-with-contracts"><a class="toclink" href="#something-more-advanced-interacting-with-contracts">Something more advanced: interacting with contracts</a></h3>
<p>We developed mockup mode as a safe environment to develop and test Michelson smart contracts.</p>
<p>To interact with a contract, we must first <a href="https://tezos.gitlab.io/007/glossary.html#origination">originate (deploy) it</a>. Let’s add a <code>dummy</code> contract to our mockup state:</p>
<p><a name="dummy-originate"></a></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>originate<span class="w"> </span>contract<span class="w"> </span>dummy<span class="w"> </span>transferring<span class="w"> </span><span class="m">100</span><span class="w"> </span>from<span class="w"> </span>bootstrap1<span class="w"> </span>running<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s1">'parameter unit; storage unit; code { CAR; NIL operation; PAIR}'</span><span class="w"> </span>--burn-cap<span class="w"> </span><span class="m">10</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="nt">Node</span><span class="w"> </span><span class="nt">is</span><span class="w"> </span><span class="nt">bootstrapped</span><span class="o">.</span>
<span class="nt">Estimated</span><span class="w"> </span><span class="nt">gas</span><span class="o">:</span><span class="w"> </span><span class="nt">1589</span><span class="p">.</span><span class="nc">562</span><span class="w"> </span><span class="nt">units</span><span class="w"> </span><span class="o">(</span><span class="nt">will</span><span class="w"> </span><span class="nt">add</span><span class="w"> </span><span class="nt">100</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nt">36</span><span class="w"> </span><span class="nt">for</span><span class="w"> </span><span class="nt">safety</span><span class="o">)</span>
<span class="nt">Estimated</span><span class="w"> </span><span class="nt">storage</span><span class="o">:</span><span class="w"> </span><span class="nt">295</span><span class="w"> </span><span class="nt">bytes</span><span class="w"> </span><span class="nt">added</span><span class="w"> </span><span class="o">(</span><span class="nt">will</span><span class="w"> </span><span class="nt">add</span><span class="w"> </span><span class="nt">20</span><span class="w"> </span><span class="nt">for</span><span class="w"> </span><span class="nt">safety</span><span class="o">)</span>
<span class="nt">Operation</span><span class="w"> </span><span class="nt">successfully</span><span class="w"> </span><span class="nt">injected</span><span class="w"> </span><span class="nt">in</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">node</span><span class="o">.</span>
<span class="nt">Operation</span><span class="w"> </span><span class="nt">hash</span><span class="w"> </span><span class="nt">is</span><span class="w"> </span><span class="s1">'oor3iMLau7g9K78pWTrAETx5KwrE7jTWYR7euh2MC6pqReV8aX7'</span>
<span class="nt">NOT</span><span class="w"> </span><span class="nt">waiting</span><span class="w"> </span><span class="nt">for</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">operation</span><span class="w"> </span><span class="nt">to</span><span class="w"> </span><span class="nt">be</span><span class="w"> </span><span class="nt">included</span><span class="o">.</span>
<span class="nt">Use</span><span class="w"> </span><span class="nt">command</span>
<span class="w"> </span><span class="nt">tezos-client</span><span class="w"> </span><span class="nt">wait</span><span class="w"> </span><span class="nt">for</span><span class="w"> </span><span class="nt">oor3iMLau7g9K78pWTrAETx5KwrE7jTWYR7euh2MC6pqReV8aX7</span><span class="w"> </span><span class="nt">to</span><span class="w"> </span><span class="nt">be</span><span class="w"> </span><span class="nt">included</span><span class="w"> </span><span class="nt">--confirmations</span><span class="w"> </span><span class="nt">30</span><span class="w"> </span><span class="nt">--branch</span><span class="w"> </span><span class="nt">BLockGenesisGenesisGenesisGenesisGenesisCCCCCeZiLHU</span>
<span class="nt">and</span><span class="o">/</span><span class="nt">or</span><span class="w"> </span><span class="nt">an</span><span class="w"> </span><span class="nt">external</span><span class="w"> </span><span class="nt">block</span><span class="w"> </span><span class="nt">explorer</span><span class="w"> </span><span class="nt">to</span><span class="w"> </span><span class="nt">make</span><span class="w"> </span><span class="nt">sure</span><span class="w"> </span><span class="nt">that</span><span class="w"> </span><span class="nt">it</span><span class="w"> </span><span class="nt">has</span><span class="w"> </span><span class="nt">been</span><span class="w"> </span><span class="nt">included</span><span class="o">.</span>
<span class="nt">This</span><span class="w"> </span><span class="nt">sequence</span><span class="w"> </span><span class="nt">of</span><span class="w"> </span><span class="nt">operations</span><span class="w"> </span><span class="nt">was</span><span class="w"> </span><span class="nt">run</span><span class="o">:</span>
<span class="w"> </span><span class="nt">Manager</span><span class="w"> </span><span class="nt">signed</span><span class="w"> </span><span class="nt">operations</span><span class="o">:</span>
<span class="w"> </span><span class="nt">From</span><span class="o">:</span><span class="w"> </span><span class="nt">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span>
<span class="w"> </span><span class="nt">Fee</span><span class="w"> </span><span class="nt">to</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">baker</span><span class="o">:</span><span class="w"> </span><span class="nt">ꜩ0</span><span class="p">.</span><span class="nc">000441</span>
<span class="w"> </span><span class="nt">Expected</span><span class="w"> </span><span class="nt">counter</span><span class="o">:</span><span class="w"> </span><span class="nt">2</span>
<span class="w"> </span><span class="nt">Gas</span><span class="w"> </span><span class="nt">limit</span><span class="o">:</span><span class="w"> </span><span class="nt">1726</span>
<span class="w"> </span><span class="nt">Storage</span><span class="w"> </span><span class="nt">limit</span><span class="o">:</span><span class="w"> </span><span class="nt">315</span><span class="w"> </span><span class="nt">bytes</span>
<span class="w"> </span><span class="nt">Balance</span><span class="w"> </span><span class="nt">updates</span><span class="o">:</span>
<span class="w"> </span><span class="nt">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span><span class="w"> </span><span class="o">................</span><span class="w"> </span><span class="nt">-ꜩ0</span><span class="p">.</span><span class="nc">000441</span>
<span class="w"> </span><span class="nt">fees</span><span class="o">(</span><span class="nt">the</span><span class="w"> </span><span class="nt">baker</span><span class="w"> </span><span class="nt">who</span><span class="w"> </span><span class="nt">will</span><span class="w"> </span><span class="nt">include</span><span class="w"> </span><span class="nt">this</span><span class="w"> </span><span class="nt">operation</span><span class="o">,</span><span class="nt">0</span><span class="o">)</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="o">+</span><span class="nt">ꜩ0</span><span class="p">.</span><span class="nc">000441</span>
<span class="w"> </span><span class="nt">Origination</span><span class="o">:</span>
<span class="w"> </span><span class="nt">From</span><span class="o">:</span><span class="w"> </span><span class="nt">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span>
<span class="w"> </span><span class="nt">Credit</span><span class="o">:</span><span class="w"> </span><span class="nt">ꜩ100</span>
<span class="w"> </span><span class="nt">Script</span><span class="o">:</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">parameter</span><span class="w"> </span><span class="err">unit</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="err">storage</span><span class="w"> </span><span class="err">unit</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="err">code</span><span class="w"> </span><span class="err">{</span><span class="w"> </span><span class="err">CAR</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="err">NIL</span><span class="w"> </span><span class="err">operation</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="err">PAIR</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="err">}</span>
<span class="w"> </span><span class="nt">Initial</span><span class="w"> </span><span class="nt">storage</span><span class="o">:</span><span class="w"> </span><span class="nt">Unit</span>
<span class="w"> </span><span class="nt">No</span><span class="w"> </span><span class="nt">delegate</span><span class="w"> </span><span class="nt">for</span><span class="w"> </span><span class="nt">this</span><span class="w"> </span><span class="nt">contract</span>
<span class="w"> </span><span class="nt">This</span><span class="w"> </span><span class="nt">origination</span><span class="w"> </span><span class="nt">was</span><span class="w"> </span><span class="nt">successfully</span><span class="w"> </span><span class="nt">applied</span>
<span class="w"> </span><span class="nt">Originated</span><span class="w"> </span><span class="nt">contracts</span><span class="o">:</span>
<span class="w"> </span><span class="nt">KT1QgvWVQHXDu6ryqQ1t3GN3UciToFbLhu7j</span>
<span class="w"> </span><span class="nt">Storage</span><span class="w"> </span><span class="nt">size</span><span class="o">:</span><span class="w"> </span><span class="nt">38</span><span class="w"> </span><span class="nt">bytes</span>
<span class="w"> </span><span class="nt">Paid</span><span class="w"> </span><span class="nt">storage</span><span class="w"> </span><span class="nt">size</span><span class="w"> </span><span class="nt">diff</span><span class="o">:</span><span class="w"> </span><span class="nt">38</span><span class="w"> </span><span class="nt">bytes</span>
<span class="w"> </span><span class="nt">Consumed</span><span class="w"> </span><span class="nt">gas</span><span class="o">:</span><span class="w"> </span><span class="nt">1589</span><span class="p">.</span><span class="nc">562</span>
<span class="w"> </span><span class="nt">Balance</span><span class="w"> </span><span class="nt">updates</span><span class="o">:</span>
<span class="w"> </span><span class="nt">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="nt">-ꜩ0</span><span class="p">.</span><span class="nc">0095</span>
<span class="w"> </span><span class="nt">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="nt">-ꜩ0</span><span class="p">.</span><span class="nc">06425</span>
<span class="w"> </span><span class="nt">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="nt">-ꜩ100</span>
<span class="w"> </span><span class="nt">KT1QgvWVQHXDu6ryqQ1t3GN3UciToFbLhu7j</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="o">+</span><span class="nt">ꜩ100</span>
<span class="nt">New</span><span class="w"> </span><span class="nt">contract</span><span class="w"> </span><span class="nt">KT1QgvWVQHXDu6ryqQ1t3GN3UciToFbLhu7j</span><span class="w"> </span><span class="nt">originated</span><span class="o">.</span>
<span class="nt">Contract</span><span class="w"> </span><span class="nt">memorized</span><span class="w"> </span><span class="nt">as</span><span class="w"> </span><span class="nt">dummy</span><span class="o">.</span><span class="w"> </span>
</code></pre></div>
<p>We now check some things.</p>
<ul>
<li>
The contract account <code>dummy</code> is now listed as known along with all bootstrap accounts:
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>list<span class="w"> </span>known<span class="w"> </span>contracts
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="n">dummy</span><span class="o">:</span><span class="w"> </span><span class="n">KT1QgvWVQHXDu6ryqQ1t3GN3UciToFbLhu7j</span>
<span class="n">bootstrap5</span><span class="o">:</span><span class="w"> </span><span class="n">tz1ddb9NMYHZi5UzPdzTZMYQQZoMub195zgv</span>
<span class="n">bootstrap4</span><span class="o">:</span><span class="w"> </span><span class="n">tz1b7tUupMgCNw2cCLpKTkSD1NZzB5TkP2sv</span>
<span class="n">bootstrap3</span><span class="o">:</span><span class="w"> </span><span class="n">tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU</span>
<span class="n">bootstrap2</span><span class="o">:</span><span class="w"> </span><span class="n">tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN</span>
<span class="n">bootstrap1</span><span class="o">:</span><span class="w"> </span><span class="n">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span>
</code></pre></div>
<li> Contract <code>dummy</code> also has the expected amount of ꜩ in its balance:
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>get<span class="w"> </span>balance<span class="w"> </span><span class="k">for</span><span class="w"> </span>dummy
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="mf">100</span><span class="w"> </span><span class="n">ꜩ</span>
</code></pre></div>
</ul>
<p>Let’s inspect our <a href="#dummy-originate">freshly-originated <code>dummy</code> contract</a> and display part of its state through its storage:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>get<span class="w"> </span>contract<span class="w"> </span>storage<span class="w"> </span><span class="k">for</span><span class="w"> </span>dummy
</code></pre></div>
<div class="highlight"><pre><span></span><code>Unit
</code></pre></div>
<p>The contract’s storage can also be accessed in <span class="caps">JSON</span> format through the
usual <span class="caps">RPC</span> mechanism:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>rpc<span class="w"> </span>get<span class="w"> </span>/chains/main/blocks/head/context/contracts/KT1QgvWVQHXDu6ryqQ1t3GN3UciToFbLhu7j/storage
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"> </span><span class="nt">"prim"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Unit"</span><span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>We can of course send some money to <code>dummy</code> and verify it has been added to the contract’s balance:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>transfer<span class="w"> </span><span class="m">100</span><span class="w"> </span>from<span class="w"> </span>bootstrap3<span class="w"> </span>to<span class="w"> </span>dummy
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="nv">Node</span><span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="nv">bootstrapped</span>.
<span class="nv">Estimated</span><span class="w"> </span><span class="nv">gas</span>:<span class="w"> </span><span class="mi">2237</span>.<span class="mi">715</span><span class="w"> </span><span class="nv">units</span><span class="w"> </span><span class="ss">(</span><span class="nv">will</span><span class="w"> </span><span class="nv">add</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">safety</span><span class="ss">)</span>
<span class="nv">Estimated</span><span class="w"> </span><span class="nv">storage</span>:<span class="w"> </span><span class="nv">no</span><span class="w"> </span><span class="nv">bytes</span><span class="w"> </span><span class="nv">added</span>
<span class="nv">Operation</span><span class="w"> </span><span class="nv">successfully</span><span class="w"> </span><span class="nv">injected</span><span class="w"> </span><span class="nv">in</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">node</span>.
<span class="nv">Operation</span><span class="w"> </span><span class="nv">hash</span><span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="s1">'ooAe9HRnc1veUPTVPBtMEpfUi5isgYm4MzeD13MN8Nxfdgfa8AZ'</span>
<span class="nv">NOT</span><span class="w"> </span><span class="nv">waiting</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">operation</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">be</span><span class="w"> </span><span class="nv">included</span>.
<span class="nv">Use</span><span class="w"> </span><span class="nv">command</span>
<span class="w"> </span><span class="nv">tezos</span><span class="o">-</span><span class="nv">client</span><span class="w"> </span><span class="k">wait</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">ooAe9HRnc1veUPTVPBtMEpfUi5isgYm4MzeD13MN8Nxfdgfa8AZ</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">be</span><span class="w"> </span><span class="nv">included</span><span class="w"> </span><span class="o">--</span><span class="nv">confirmations</span><span class="w"> </span><span class="mi">30</span><span class="w"> </span><span class="o">--</span><span class="nv">branch</span><span class="w"> </span><span class="nv">BLockGenesisGenesisGenesisGenesisGenesisCCCCCeZiLHU</span>
<span class="nv">and</span><span class="o">/</span><span class="nv">or</span><span class="w"> </span><span class="nv">an</span><span class="w"> </span><span class="nv">external</span><span class="w"> </span><span class="nv">block</span><span class="w"> </span><span class="nv">explorer</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">make</span><span class="w"> </span><span class="nv">sure</span><span class="w"> </span><span class="nv">that</span><span class="w"> </span><span class="nv">it</span><span class="w"> </span><span class="nv">has</span><span class="w"> </span><span class="nv">been</span><span class="w"> </span><span class="nv">included</span>.
<span class="nv">This</span><span class="w"> </span><span class="nv">sequence</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">operations</span><span class="w"> </span><span class="nv">was</span><span class="w"> </span><span class="nv">run</span>:
<span class="w"> </span><span class="nv">Manager</span><span class="w"> </span><span class="nv">signed</span><span class="w"> </span><span class="nv">operations</span>:
<span class="w"> </span><span class="nv">From</span>:<span class="w"> </span><span class="nv">tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU</span>
<span class="w"> </span><span class="nv">Fee</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">baker</span>:<span class="w"> </span>ꜩ<span class="mi">0</span>.<span class="mi">000485</span>
<span class="w"> </span><span class="nv">Expected</span><span class="w"> </span><span class="nv">counter</span>:<span class="w"> </span><span class="mi">1</span>
<span class="w"> </span><span class="nv">Gas</span><span class="w"> </span><span class="nv">limit</span>:<span class="w"> </span><span class="mi">2338</span>
<span class="w"> </span><span class="nv">Storage</span><span class="w"> </span><span class="nv">limit</span>:<span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="nv">bytes</span>
<span class="w"> </span><span class="nv">Balance</span><span class="w"> </span><span class="nv">updates</span>:
<span class="w"> </span><span class="nv">tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU</span><span class="w"> </span>................<span class="w"> </span><span class="o">-</span>ꜩ<span class="mi">0</span>.<span class="mi">000485</span>
<span class="w"> </span><span class="nv">fees</span><span class="ss">(</span><span class="nv">the</span><span class="w"> </span><span class="nv">baker</span><span class="w"> </span><span class="nv">who</span><span class="w"> </span><span class="nv">will</span><span class="w"> </span><span class="k">include</span><span class="w"> </span><span class="nv">this</span><span class="w"> </span><span class="nv">operation</span>,<span class="mi">0</span><span class="ss">)</span><span class="w"> </span>...<span class="w"> </span><span class="o">+</span>ꜩ<span class="mi">0</span>.<span class="mi">000485</span>
<span class="w"> </span><span class="nv">Transaction</span>:
<span class="w"> </span><span class="nv">Amount</span>:<span class="w"> </span>ꜩ<span class="mi">100</span>
<span class="w"> </span><span class="nv">From</span>:<span class="w"> </span><span class="nv">tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU</span>
<span class="w"> </span><span class="nv">To</span>:<span class="w"> </span><span class="nv">KT1QgvWVQHXDu6ryqQ1t3GN3UciToFbLhu7j</span>
<span class="w"> </span><span class="nv">This</span><span class="w"> </span><span class="nv">transaction</span><span class="w"> </span><span class="nv">was</span><span class="w"> </span><span class="nv">successfully</span><span class="w"> </span><span class="nv">applied</span>
<span class="w"> </span><span class="nv">Updated</span><span class="w"> </span><span class="nv">storage</span>:<span class="w"> </span><span class="nv">Unit</span>
<span class="w"> </span><span class="nv">Storage</span><span class="w"> </span><span class="nv">size</span>:<span class="w"> </span><span class="mi">38</span><span class="w"> </span><span class="nv">bytes</span>
<span class="w"> </span><span class="nv">Consumed</span><span class="w"> </span><span class="nv">gas</span>:<span class="w"> </span><span class="mi">2237</span>.<span class="mi">715</span>
<span class="w"> </span><span class="nv">Balance</span><span class="w"> </span><span class="nv">updates</span>:
<span class="w"> </span><span class="nv">tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU</span><span class="w"> </span>...<span class="w"> </span><span class="o">-</span>ꜩ<span class="mi">100</span>
<span class="w"> </span><span class="nv">KT1QgvWVQHXDu6ryqQ1t3GN3UciToFbLhu7j</span><span class="w"> </span>...<span class="w"> </span><span class="o">+</span>ꜩ<span class="mi">100</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>get<span class="w"> </span>balance<span class="w"> </span><span class="k">for</span><span class="w"> </span>bootstrap3
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="mf">3999899.999515</span><span class="w"> </span><span class="n">ꜩ</span>
<span class="err">````</span>
<span class="err">```</span><span class="n">shell</span>
<span class="err">$</span><span class="w"> </span><span class="n">mockup</span><span class="o">-</span><span class="n">client</span><span class="w"> </span><span class="kr">get</span><span class="w"> </span><span class="n">balance</span><span class="w"> </span><span class="kr">for</span><span class="w"> </span><span class="n">dummy</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="mf">200</span><span class="w"> </span><span class="n">ꜩ</span>
</code></pre></div>
<p>The examples so far have used mockup mode’s default settings. Some use cases need a custom setup, so mockup mode lets us configure some initial parameters, as we discuss next.</p>
<h2 id="tune-mockup-parameters"><a class="toclink" href="#tune-mockup-parameters">Tune mockup parameters</a></h2>
<p>For simplicity, mockup mode — like sandboxed mode — uses default
values for wallet and protocol parameters. These default settings can be
inspected and overridden to suit your needs.</p>
<p>The default configuration can be inspected as follows
(<a href="#alias_definition">recall that <code>mockup-client</code> is a command alias for <code>tezos-client</code> plus some parameters</a>):</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>config<span class="w"> </span>show
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="err">De</span><span class="kc">fault</span><span class="w"> </span><span class="err">value</span><span class="w"> </span><span class="err">o</span><span class="kc">f</span><span class="w"> </span><span class="mi">--</span><span class="err">boo</span><span class="kc">tstra</span><span class="err">p</span><span class="mi">-</span><span class="err">accou</span><span class="kc">nts</span><span class="p">:</span>
<span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"bootstrap1"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"sk_uri"</span><span class="p">:</span>
<span class="w"> </span><span class="s2">"unencrypted:edsk3gUfUPyBSfrS9CCgmCiQsTCHGkviBDusMxDJstFtojtc1zcpsh"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"amount"</span><span class="p">:</span><span class="w"> </span><span class="s2">"3999799925405"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"bootstrap2"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"sk_uri"</span><span class="p">:</span>
<span class="w"> </span><span class="s2">"unencrypted:edsk39qAm1fiMjgmPkw1EgQYkMzkJezLNewd7PLNHTkr6w9XA2zdfo"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"amount"</span><span class="p">:</span><span class="w"> </span><span class="s2">"4000100000000"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"bootstrap3"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"sk_uri"</span><span class="p">:</span>
<span class="w"> </span><span class="s2">"unencrypted:edsk4ArLQgBTLWG5FJmnGnT689VKoqhXwmDPBuGx3z4cvwU9MmrPZZ"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"amount"</span><span class="p">:</span><span class="w"> </span><span class="s2">"3999899999515"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"bootstrap4"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"sk_uri"</span><span class="p">:</span>
<span class="w"> </span><span class="s2">"unencrypted:edsk2uqQB9AY4FvioK2YMdfmyMrer5R8mGFyuaLLFfSRo8EoyNdht3"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"amount"</span><span class="p">:</span><span class="w"> </span><span class="s2">"4000000000000"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"bootstrap5"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"sk_uri"</span><span class="p">:</span>
<span class="w"> </span><span class="s2">"unencrypted:edsk4QLrcijEffxV31gGdN2HU7UpyJjA8drFoNcmnB28n89YjPNRFm"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"amount"</span><span class="p">:</span><span class="w"> </span><span class="s2">"4000000000000"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span>
<span class="err">De</span><span class="kc">fault</span><span class="w"> </span><span class="err">value</span><span class="w"> </span><span class="err">o</span><span class="kc">f</span><span class="w"> </span><span class="mi">--</span><span class="err">pro</span><span class="kc">t</span><span class="err">ocol</span><span class="mi">-</span><span class="err">co</span><span class="kc">nstants</span><span class="p">:</span>
<span class="p">{</span><span class="w"> </span><span class="nt">"hard_gas_limit_per_operation"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1040000"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"hard_gas_limit_per_block"</span><span class="p">:</span><span class="w"> </span><span class="s2">"10400000"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"hard_storage_limit_per_operation"</span><span class="p">:</span><span class="w"> </span><span class="s2">"60000"</span><span class="p">,</span><span class="w"> </span><span class="nt">"cost_per_byte"</span><span class="p">:</span><span class="w"> </span><span class="s2">"250"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"chain_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"NetXynUjJNZm7wi"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"initial_timestamp"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1970-01-01T00:00:00Z"</span><span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>We can tune these values with dedicated mockup mode creation switches. Generating the <span class="caps">JSON</span> data by hand is a pain (and dangerously error-prone) so we suggest to generate files corresponding to default values, and edit them.</p>
<p>To generate the <span class="caps">JSON</span> files related to protocol constants and bootstrap accounts
configuration, just type:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>config<span class="w"> </span>init
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="n">Written</span><span class="w"> </span><span class="n">default</span><span class="w"> </span><span class="o">--</span><span class="n">bootstrap</span><span class="o">-</span><span class="n">accounts</span><span class="w"> </span><span class="n">file</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">mockup</span><span class="o">/</span><span class="n">bootstrap</span><span class="o">-</span><span class="n">accounts</span><span class="o">.</span><span class="n">json</span>
<span class="n">Written</span><span class="w"> </span><span class="n">default</span><span class="w"> </span><span class="o">--</span><span class="n">protocol</span><span class="o">-</span><span class="n">constants</span><span class="w"> </span><span class="n">file</span><span class="p">:</span><span class="w"> </span><span class="o">/</span><span class="n">tmp</span><span class="o">/</span><span class="n">mockup</span><span class="o">/</span><span class="n">protocol</span><span class="o">-</span><span class="n">constants</span><span class="o">.</span><span class="n">json</span>
</code></pre></div>
<p>We can now edit the files <code>bootstrap-accounts.json</code> and
<code>protocol-constants.json</code> to later create a tuned mockup state.</p>
<p>For example, we can change the <code>chain_id</code> field of <code>protocol-constants.json</code>.
We will compute a new chain identifier, which will replace the initial
<code>NetXynUjJNZm7wi</code> value.</p>
<p><a name="compute_chain_id"></a></p>
<div class="highlight"><pre><span></span><code>tezos-client<span class="w"> </span>compute<span class="w"> </span>chain<span class="w"> </span>id<span class="w"> </span>from<span class="w"> </span>seed<span class="w"> </span>my-chain-id
</code></pre></div>
<div class="highlight"><pre><span></span><code>NetXKQNvsbETtvZ
</code></pre></div>
<p>Let’s create a new protocol constants configuration file, using <code>jq</code> to
manipulate the <span class="caps">JSON</span> data.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>/tmp/mockup/protocol-constants.json<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>jq<span class="w"> </span><span class="s1">'.chain_id = "NetXKQNvsbETtvZ"'</span><span class="w"> </span>><span class="w"> </span>tuned_up_protocol_constants.json
</code></pre></div>
<p>Assuming you have not renamed the files, you can create a new mockup setup by
feeding the <span class="caps">JSON</span> configuration to the <code>create mockup</code> command with the following
command-line invocation.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mv<span class="w"> </span>/tmp/mockup<span class="w"> </span>/tmp/mockup.old<span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>mockup-client<span class="w"> </span>--protocol<span class="w"> </span>ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>create<span class="w"> </span>mockup<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--protocol-constants<span class="w"> </span>tuned_up_protocol_constants.json<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--bootstrap-accounts<span class="w"> </span>/tmp/mockup.old/bootstrap-accounts.json
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="nt">Created</span><span class="w"> </span><span class="nt">mockup</span><span class="w"> </span><span class="nt">client</span><span class="w"> </span><span class="nt">base</span><span class="w"> </span><span class="nt">dir</span><span class="w"> </span><span class="nt">in</span><span class="w"> </span><span class="o">/</span><span class="nt">tmp</span><span class="o">/</span><span class="nt">mockup</span>
<span class="nt">mockup</span><span class="w"> </span><span class="nt">client</span><span class="w"> </span><span class="nt">uses</span><span class="w"> </span><span class="nt">protocol</span><span class="w"> </span><span class="nt">overrides</span><span class="o">:</span>
<span class="nt">hard_gas_limit_per_operation</span><span class="o">:</span><span class="w"> </span><span class="nt">1040000</span>
<span class="nt">hard_gas_limit_per_block</span><span class="o">:</span><span class="w"> </span><span class="nt">10400000</span>
<span class="nt">hard_storage_limit_per_operation</span><span class="o">:</span><span class="w"> </span><span class="nt">60000</span>
<span class="nt">cost_per_byte</span><span class="o">:</span><span class="w"> </span><span class="nt">0</span><span class="p">.</span><span class="nc">00025</span>
<span class="nt">mockup</span><span class="w"> </span><span class="nt">client</span><span class="w"> </span><span class="nt">uses</span><span class="w"> </span><span class="nt">custom</span><span class="w"> </span><span class="nt">bootstrap</span><span class="w"> </span><span class="nt">accounts</span><span class="o">:</span>
<span class="nt">name</span><span class="p">:</span><span class="nd">bootstrap1</span>
<span class="nt">sk_uri</span><span class="p">:</span><span class="nd">unencrypted</span><span class="p">:</span><span class="nd">edsk3gUfUPyBSfrS9CCgmCiQsTCHGkviBDusMxDJstFtojtc1zcpsh</span>
<span class="nt">amount</span><span class="p">:</span><span class="nd">3999799</span><span class="p">.</span><span class="nc">925405</span><span class="o">;</span>
<span class="nt">name</span><span class="p">:</span><span class="nd">bootstrap2</span>
<span class="nt">sk_uri</span><span class="p">:</span><span class="nd">unencrypted</span><span class="p">:</span><span class="nd">edsk39qAm1fiMjgmPkw1EgQYkMzkJezLNewd7PLNHTkr6w9XA2zdfo</span>
<span class="nt">amount</span><span class="p">:</span><span class="nd">4000100</span><span class="o">;</span>
<span class="nt">name</span><span class="p">:</span><span class="nd">bootstrap3</span>
<span class="nt">sk_uri</span><span class="p">:</span><span class="nd">unencrypted</span><span class="p">:</span><span class="nd">edsk4ArLQgBTLWG5FJmnGnT689VKoqhXwmDPBuGx3z4cvwU9MmrPZZ</span>
<span class="nt">amount</span><span class="p">:</span><span class="nd">3999899</span><span class="p">.</span><span class="nc">999515</span><span class="o">;</span>
<span class="nt">name</span><span class="p">:</span><span class="nd">bootstrap4</span>
<span class="nt">sk_uri</span><span class="p">:</span><span class="nd">unencrypted</span><span class="p">:</span><span class="nd">edsk2uqQB9AY4FvioK2YMdfmyMrer5R8mGFyuaLLFfSRo8EoyNdht3</span>
<span class="nt">amount</span><span class="p">:</span><span class="nd">4000000</span><span class="o">;</span>
<span class="nt">name</span><span class="p">:</span><span class="nd">bootstrap5</span>
<span class="nt">sk_uri</span><span class="p">:</span><span class="nd">unencrypted</span><span class="p">:</span><span class="nd">edsk4QLrcijEffxV31gGdN2HU7UpyJjA8drFoNcmnB28n89YjPNRFm</span>
<span class="nt">amount</span><span class="p">:</span><span class="nd">4000000</span>
<span class="nt">Tezos</span><span class="w"> </span><span class="nt">address</span><span class="w"> </span><span class="nt">added</span><span class="o">:</span><span class="w"> </span><span class="nt">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span>
<span class="nt">Tezos</span><span class="w"> </span><span class="nt">address</span><span class="w"> </span><span class="nt">added</span><span class="o">:</span><span class="w"> </span><span class="nt">tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN</span>
<span class="nt">Tezos</span><span class="w"> </span><span class="nt">address</span><span class="w"> </span><span class="nt">added</span><span class="o">:</span><span class="w"> </span><span class="nt">tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU</span>
<span class="nt">Tezos</span><span class="w"> </span><span class="nt">address</span><span class="w"> </span><span class="nt">added</span><span class="o">:</span><span class="w"> </span><span class="nt">tz1b7tUupMgCNw2cCLpKTkSD1NZzB5TkP2sv</span>
<span class="nt">Tezos</span><span class="w"> </span><span class="nt">address</span><span class="w"> </span><span class="nt">added</span><span class="o">:</span><span class="w"> </span><span class="nt">tz1ddb9NMYHZi5UzPdzTZMYQQZoMub195z</span>
</code></pre></div>
<p>We can check that the chain id in our environment setup matches <a href="#compute_chain_id">the chain id we obtained from the command line</a>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>/tmp/mockup/mockup/context.json<span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.chain_id
</code></pre></div>
<div class="highlight"><pre><span></span><code>"NetXKQNvsbETtvZ"
</code></pre></div>
<h3 id="context-state"><a class="toclink" href="#context-state">Context state</a></h3>
<p>In addition to the two <code>bootstrap-accounts.json</code> and
<code>protocol-constants.json</code> configuration files, stateful mockup
mode stores state data in a single <code>context.json</code> file. </p>
<p><code>context.json</code> is located under
the <code>mockup</code> subdirectory of the base directory. In our running example, its
absolute file name is <code>/tmp/mockup/mockup/context.json</code>.
It contains in particular information about the current block hash or the shell header:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>/tmp/mockup/mockup/context.json<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>jq<span class="w"> </span><span class="s1">'.context | { block_hash: .block_hash, shell_header: .shell_header}'</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">"block_hash"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BLockGenesisGenesisGenesisGenesisGenesisCCCCCeZiLHU"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"shell_header"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">"level"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"proto"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"predecessor"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BLockGenesisGenesisGenesisGenesisGenesisCCCCCeZiLHU"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"timestamp"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1970-01-01T00:00:00Z"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"validation_pass"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"operations_hash"</span><span class="p">:</span><span class="w"> </span><span class="s2">"LLoZKi1iMzbeJrfrGWPFYmkLebcsha6vGskQ4rAXt2uMwQtBfRcjL"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"fitness"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="s2">"01"</span><span class="p">,</span>
<span class="w"> </span><span class="s2">"0000000000000000"</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"context"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CoUeJrcPBj3T3iJL3PY4jZHnmZa5rRZ87VQPdSBNBcwZRMWJGh9j"</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>The directory where the context state resides, <code>/tmp/mockup</code> in our examples, is
where mockup mode keeps all its data. In particular, it is key to supporting
asynchronous operations in stateful mockup mode.</p>
<h2 id="running-a-mockup-client-with-asynchronous-state"><a class="toclink" href="#running-a-mockup-client-with-asynchronous-state">Running a mockup client with asynchronous state <a name="asynchronous_mockup"></a></a></h2>
<p>In Tezos, extending the blockchain is a three-step process:</p>
<ol>
<li>An operation is emitted across the network of nodes.</li>
<li>It gets validated, aggregated with other operations, and included in (baked in to) a block by a baker.</li>
<li>The (cryptographic hash of the) block gets included in the next block. </li>
</ol>
<p>See <a href="https://arxiv.org/pdf/1909.08458.pdf">this paper</a> for details (search for <em>“In order to append transactions to the ledger, all blockchains follow a similar generic algorithm …”</em>).</p>
<p>Thus, mockup mode offers a <strong>stateful asynchronous mode</strong> which simulates a two-step inclusion of operations in the Tezos chain which corresponds to steps 2 and 3 above.<sup id="fnref:irrelevant"><a class="footnote-ref" href="#fn:irrelevant">6</a></sup> </p>
<p>We must add two new files in the <code>mockup</code> subdirectory, to store:</p>
<ul>
<li>operations waiting to be baked in (<code>mempool.json</code>) and </li>
<li>operations rejected (<code>trashpool.json</code>). </li>
</ul>
<h3 id="how-to-activate"><a class="toclink" href="#how-to-activate">How to activate</a></h3>
<p>To activate <em>asynchronous</em> stateful mockup mode, we reuse the <a href="#making_the_state_command">initial
command line invocation for state creation</a>, with an <code>--asynchronous</code>
flag:</p>
<p><a name="asynchronous_mockup_example"></a></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>rm<span class="w"> </span>-Rf<span class="w"> </span>/tmp/mockup<span class="w"> </span><span class="o">&&</span><span class="w"> </span>mockup-client<span class="w"> </span>create<span class="w"> </span>mockup<span class="w"> </span>--asynchronous
</code></pre></div>
<div class="highlight"><pre><span></span><code>Created mockup client base dir in /tmp/mockup
creating mempool file at /tmp/mockup/mockup/mempool.json
Tezos address added: tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx
Tezos address added: tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN
Tezos address added: tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU
Tezos address added: tz1b7tUupMgCNw2cCLpKTkSD1NZzB5TkP2sv
Tezos address added: tz1ddb9NMYHZi5UzPdzTZMYQQZoMub195zgv
</code></pre></div>
<p>This commands creates a fresh mockup directory, as for <a href="#stateful_mockup">stateful mode</a>, but adds another file to represent the mempool (<code>mempool.json</code>), which is
initially empty.</p>
<h3 id="baking-in-asynchronous-stateful-mockup-mode"><a class="toclink" href="#baking-in-asynchronous-stateful-mockup-mode">Baking in asynchronous stateful mockup mode</a></h3>
<p>Let’s add some operations to <code>mempool.json</code> by issuing two transfers.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>transfer<span class="w"> </span><span class="m">1</span><span class="w"> </span>from<span class="w"> </span>bootstrap1<span class="w"> </span>to<span class="w"> </span>bootstrap2
$<span class="w"> </span>mockup-client<span class="w"> </span>transfer<span class="w"> </span><span class="m">2</span><span class="w"> </span>from<span class="w"> </span>bootstrap2<span class="w"> </span>to<span class="w"> </span>bootstrap3
</code></pre></div>
<p>These commands use the same syntax as for <a href="#stateful-mockup-first-transfer">immediate stateful mockup mode</a>;
the fact that we are operating in asynchronous mode is auto-detected.
Both transfer operations are now in the mempool, as we can verify:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>/tmp/mockup/mockup/mempool.json
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"shell_header"</span><span class="p">:</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"branch"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BLockGenesisGenesisGenesisGenesisGenesisCCCCCeZiLHU"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">"protocol_data"</span><span class="p">:</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"contents"</span><span class="p">:</span>
<span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"kind"</span><span class="p">:</span><span class="w"> </span><span class="s2">"transaction"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"source"</span><span class="p">:</span><span class="w"> </span><span class="s2">"tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx"</span><span class="p">,</span><span class="w"> </span><span class="nt">"fee"</span><span class="p">:</span><span class="w"> </span><span class="s2">"403"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"counter"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w"> </span><span class="nt">"gas_limit"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1527"</span><span class="p">,</span><span class="w"> </span><span class="nt">"storage_limit"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"amount"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1000000"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"destination"</span><span class="p">:</span><span class="w"> </span><span class="s2">"tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"signature"</span><span class="p">:</span>
<span class="w"> </span><span class="s2">"sigTXV77JT5t3xaAUnCXs4RhvwJscFaqpZvHp4Wm8tQoENKXyFz3hLyqbQkibPoo4JNeXiGHJRdeMTAK79ZJJMDTvxZGF75H"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"shell_header"</span><span class="p">:</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"branch"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BLockGenesisGenesisGenesisGenesisGenesisCCCCCeZiLHU"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">"protocol_data"</span><span class="p">:</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"contents"</span><span class="p">:</span>
<span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"kind"</span><span class="p">:</span><span class="w"> </span><span class="s2">"transaction"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"source"</span><span class="p">:</span><span class="w"> </span><span class="s2">"tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN"</span><span class="p">,</span><span class="w"> </span><span class="nt">"fee"</span><span class="p">:</span><span class="w"> </span><span class="s2">"403"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"counter"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1"</span><span class="p">,</span><span class="w"> </span><span class="nt">"gas_limit"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1527"</span><span class="p">,</span><span class="w"> </span><span class="nt">"storage_limit"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"amount"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2000000"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"destination"</span><span class="p">:</span><span class="w"> </span><span class="s2">"tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"signature"</span><span class="p">:</span>
<span class="w"> </span><span class="s2">"sigNDKvCy4JDjSSXNG3CBwwhkFw2S8DeVaUXamYbVddsyRwZhF7rf1RgCZUZBy8UdsYrWFUDaj9b8xH3w5Ryg3SoBfGmKJBR"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span>
</code></pre></div>
<p>The mempool is just a <span class="caps">JSON</span> array listing the valid operations that are waiting to be baked in. We see our two pending transfers above, e.g. <code>transfer 1</code> is first and will earn whoever bakes it into the chain <code>403</code> microtez (0.000403 ꜩ); we will return to this fee <a href="#the-trashpool">later</a>.</p>
<p>We can check that our transfers have not yet been included in the chain (baked), so that the balances of <code>bootstrap1</code> and <code>bootstrap2</code> are unchanged. Both still have the initial value of 4000000 ꜩ:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>get<span class="w"> </span>balance<span class="w"> </span><span class="k">for</span><span class="w"> </span>bootstrap1
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="mf">4000000</span><span class="w"> </span><span class="n">ꜩ</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>get<span class="w"> </span>balance<span class="w"> </span><span class="k">for</span><span class="w"> </span>bootstrap2
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="mf">4000000</span><span class="w"> </span><span class="n">ꜩ</span>
</code></pre></div>
<p>For the balances to change, the transfers must be validated and baked
in the chain. </p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>bake<span class="w"> </span><span class="k">for</span><span class="w"> </span>bootstrap1<span class="w"> </span>--minimal-timestamp
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="nv">Nov</span><span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="mi">10</span>:<span class="mi">23</span>:<span class="mi">05</span>.<span class="mi">436</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nv">alpha</span>.<span class="nv">baking</span>.<span class="nv">forge</span>:<span class="w"> </span><span class="nv">found</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="nv">valid</span><span class="w"> </span><span class="nv">operations</span><span class="w"> </span><span class="ss">(</span><span class="mi">0</span><span class="w"> </span><span class="nv">refused</span><span class="ss">)</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">timestamp</span><span class="w"> </span><span class="mi">1970</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">01</span><span class="nv">T00</span>:<span class="mi">00</span>:<span class="mi">02</span>.<span class="mi">000</span><span class="o">-</span><span class="mi">00</span>:<span class="mi">00</span><span class="w"> </span><span class="ss">(</span><span class="nv">fitness</span><span class="w"> </span><span class="mi">01</span>::<span class="mi">0000000000000001</span><span class="ss">)</span>
<span class="nv">Injected</span><span class="w"> </span><span class="nv">block</span><span class="w"> </span><span class="nv">BLzVCvEvPRsc</span>
</code></pre></div>
<p>Note the <code>--minimal-timestamp</code> flag above. This will compute the baked block’s timestamp from its predecessor’s, instead of taking the current machine time.</p>
<p>In a local simulated environment, this also ensures the
baking action will succeed, since the (computed) time between timestamps is
guaranteed greater than the chain’s minimal time interval between
blocks. Without the <code>--minimal-timestamp</code> flag, baking might fail because it is too close
to the last baking.</p>
<p>The mempool is now empty since all operations were valid for the baking operation.<sup id="fnref:forging"><a class="footnote-ref" href="#fn:forging">7</a></sup></p>
<h3 id="the-trashpool"><a class="toclink" href="#the-trashpool">The trashpool</a></h3>
<p>The <code>trashpool.json</code> file which we will see below is a design feature unique to mockup mode.
It does not appear in normal or <a href="#differences-from-sandboxed-mode">sandboxed mode</a>.
First, some motivation:</p>
<p>The fee for <a href="#baking-in-asynchronous-stateful-mockup-mode"><code>transfer 1</code> above</a> was a default fee which in a real system would have been paid to the baker baking the transaction into the real chain.
But suppose our transfer is urgent and we want to offer an extra fee to encourage it to be baked quickly.
<code>mockup-mode</code> allows us to offer an additional incentive to our (mock) bakers:</p>
<p><a name="valid_trashed_operation"></a></p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>transfer<span class="w"> </span><span class="m">1</span><span class="w"> </span>from<span class="w"> </span>bootstrap1<span class="w"> </span>to<span class="w"> </span>bootstrap2<span class="w"> </span>--fee<span class="w"> </span><span class="m">1</span><span class="w"> </span>
$<span class="w"> </span>mockup-client<span class="w"> </span>transfer<span class="w"> </span><span class="m">2</span><span class="w"> </span>from<span class="w"> </span>bootstrap2<span class="w"> </span>to<span class="w"> </span>bootstrap3<span class="w"> </span>--fee<span class="w"> </span><span class="m">0</span>.5<span class="w"> </span>
</code></pre></div>
<p>Thus we have asked <code>mockup-client</code> to carry out two transactions: <code>transfer 1</code> with a fee of 1 ꜩ, and <code>transfer 2</code> with a fee of 0.5ꜩ.
We then execute a selective baking operation:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mockup-client<span class="w"> </span>bake<span class="w"> </span><span class="k">for</span><span class="w"> </span>bootstrap1<span class="w"> </span>--minimal-timestamp<span class="w"> </span>--minimal-fees<span class="w"> </span><span class="m">0</span>.6
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="nv">Nov</span><span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="mi">10</span>:<span class="mi">23</span>:<span class="mi">06</span>.<span class="mi">047</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nv">alpha</span>.<span class="nv">baking</span>.<span class="nv">forge</span>:<span class="w"> </span><span class="nv">found</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="nv">valid</span><span class="w"> </span><span class="nv">operations</span><span class="w"> </span><span class="ss">(</span><span class="mi">1</span><span class="w"> </span><span class="nv">refused</span><span class="ss">)</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">timestamp</span><span class="w"> </span><span class="mi">1970</span><span class="o">-</span><span class="mi">01</span><span class="o">-</span><span class="mi">01</span><span class="nv">T00</span>:<span class="mi">00</span>:<span class="mi">04</span>.<span class="mi">000</span><span class="o">-</span><span class="mi">00</span>:<span class="mi">00</span><span class="w"> </span><span class="ss">(</span><span class="nv">fitness</span><span class="w"> </span><span class="mi">01</span>::<span class="mi">0000000000000002</span><span class="ss">)</span>
<span class="nv">Nov</span><span class="w"> </span><span class="mi">9</span><span class="w"> </span><span class="mi">10</span>:<span class="mi">23</span>:<span class="mi">06</span>.<span class="mi">182</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nv">mockup</span>.<span class="nv">local_services</span>:<span class="w"> </span><span class="nv">Appending</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="nv">operation</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">trashpool</span>
<span class="nv">Injected</span><span class="w"> </span><span class="nv">block</span><span class="w"> </span><span class="nv">BLE4cu7Usm8J</span>
</code></pre></div>
<p><code>transfer 1</code> and <code>transfer 2</code> are both valid transactions with respect to the emitters’ balances, but our mock baker </p>
<ul>
<li>accepts <code>transfer 1</code> and </li>
<li>rejects <code>transfer 2</code> because of the command line option <code>--minimal-fees 0.6</code>.</li>
</ul>
<p><code>mempool.json</code> is empty after the baking operation, and the rejected <code>transaction 2</code> has been pushed into<br>
a <code>trashpool.json</code> file, for debugging:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>/tmp/mockup/mockup/mempool.json
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="p">[]</span>
</code></pre></div>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>/tmp/mockup/mockup/trashpool.json
</code></pre></div>
<div class="highlight"><pre><span></span><code><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"shell_header"</span><span class="p">:</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"branch"</span><span class="p">:</span><span class="w"> </span><span class="s2">"BLzVCvEvPRscvX9jre4t9oLWcSMa5o5LfnhPtSGsiL45qUSLRcB"</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">"protocol_data"</span><span class="p">:</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"contents"</span><span class="p">:</span>
<span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">"kind"</span><span class="p">:</span><span class="w"> </span><span class="s2">"transaction"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"source"</span><span class="p">:</span><span class="w"> </span><span class="s2">"tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"fee"</span><span class="p">:</span><span class="w"> </span><span class="s2">"500000"</span><span class="p">,</span><span class="w"> </span><span class="nt">"counter"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2"</span><span class="p">,</span><span class="w"> </span><span class="nt">"gas_limit"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1527"</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="nt">"storage_limit"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0"</span><span class="p">,</span><span class="w"> </span><span class="nt">"amount"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2000000"</span><span class="p">,</span>
<span class="w"> </span><span class="nt">"destination"</span><span class="p">:</span><span class="w"> </span><span class="s2">"tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">"signature"</span><span class="p">:</span>
<span class="w"> </span><span class="s2">"sigpzoMG3ySUrk3JvVhjWgzK86NXbUdvD4Zad84GnC3e75ZxoAtMJAUe2tPa4UtFcr1cqmruruewYC4r9nWfKmVZcLSTaPZp"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span>
</code></pre></div>
<p>The trashpool behaves like (and indeed is) a debug log: once an operation is recorded in the trashpool, this log entry stays there forever and cannot leave. This is because mockup mode has only one head block and one chain, and we do not store information about the past beyond the head block and the block preceding it. This is consistent with an idea of mockup mode as an environment for a single user to test state transformations on a single machine. </p>
<h3 id="differences-from-sandboxed-mode"><a class="toclink" href="#differences-from-sandboxed-mode">Differences from sandboxed mode</a></h3>
<p>Mockup mode is not the only way to safely and locally test a Tezos blockchain: we can also run a Tezos blockchain <a href="https://tezos.gitlab.io/user/sandbox.html">sandboxed</a> on a local machine. So in particular we can use sandboxed mode to create a blockchain consisting of a single Tezos node, and then interact with that using <code>tezos-client</code>.</p>
<p>This is a more heavyweight solution, because a sandboxed blockchain is still a blockchain,
albeit one that is isolated (sandboxed) from the wider internet.
A sandboxed blockchain is just as complex as an unsandboxed chain: it comes with chain history,<sup id="fnref:history"><a class="footnote-ref" href="#fn:history">8</a></sup> chain branching, consensus, and so forth. These are all good things, but they are not needed for every testing scenario.</p>
<p>Sandboxed mode is also somewhat less friendly to debugging, e.g. it has no trashpool.</p>
<p>So, while it is valid to run a sandboxed blockchain and examine its behavior — and this has been done in practice — it requires some effort.</p>
<p>In mockup mode, in contrast, </p>
<ul>
<li>there is no live node, </li>
<li>there is no chain branching (there is only ever <em>one</em> live block: the head), and </li>
<li>there is almost no history (we just store two blocks). </li>
</ul>
<p>But, </p>
<ul>
<li>Mockup mode is lightweight, fast, and convenient, and </li>
<li>there is no networking overhead, and no rounds of consensus with the one available node to decide which of the one available blocks will be added to the one available chain. </li>
<li>Mockup mode gives direct command-line access to some critical helper functions of a full Tezos implementation, allowing us to run and test those functions acting directly on a locally stored state. </li>
</ul>
<p>Thus workflow is simple. The initial state, once created, is directly accessible and transformable — no need to open a terminal, run a node, log into clients, communicate with the node, let the node communicate (with itself) to reach consensus (with itself), and so forth. </p>
<h2 id="conclusions"><a class="toclink" href="#conclusions">Conclusions</a></h2>
<p>We have presented a general overview of mockup mode in its <a href="#three-modes-of-operation">three modes of operation</a>: </p>
<ul>
<li><a href="#stateless_mockup">Stateless</a> mode gives us access to some basic but important functions. </li>
<li><a href="#stateful_mockup">Stateful</a> mode gives us a state, but no baking. Every operation is immediately acted on by either being registered in the state, or rejected. There is only one live block. </li>
<li><a href="#asynchronous_mockup">Stateful asynchronous</a> adds baking. There are a mempool and trashpool, and baking operations from the mempool either to act on the local state or to get dropped into the trashpool. There is still only one live block. </li>
</ul>
<p>If you want to go beyond this then you can either:</p>
<ul>
<li>set up a <a href="https://tezos.gitlab.io/user/sandbox.html">sandboxed</a> Tezos network, </li>
<li>set up your own live Tezos network, or </li>
<li>connect to one that somebody else has already set up (e.g. <em>the</em> Tezos Mainnet).</li>
</ul>
<p>Thus mockup mode fills in a complete menu of options for experimenting with Tezos.</p>
<p>We created this new version of <code>tezos-client</code>, with its mockup mode, to help our developers to quickly and efficiently develop and test smart contracts in a Tezos environment. They found it useful and it has improved our internal development cycle.<sup id="fnref:internal"><a class="footnote-ref" href="#fn:internal">9</a></sup> We are happy to share this tool with the Tezos community, and we hope you will like it and find it useful too.</p>
<p>Mockup mode is being actively developed and will evolve best if it can benefit from your feedback. If you have a suggestion, please do not hesitate to create an issue on <a href="https://gitlab.com/tezos/tezos/-/issues">the tezos issue tracker</a>.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:versions">
<p><em>Technical note:</em> The <a href="https://gitlab.com/tezos/tezos/-/tree/v7-release">current version <code>7.x</code> releases</a> have <a href="https://gitlab.com/tezos/tezos/-/tree/v7-release/src/lib_mockup">a preliminary version</a>, with a slightly different user experience. Mockup mode has existed for Tezos protocols starting with <code>Carthage</code> (numbers relates to shell updates, and names to protocol updates). Edo requires a shell containing <a href="https://tezos.gitlab.io/protocols/008_edo.html#environment-v1">environment V1</a>. Delphi and Carthage are both usable on 7.x releases, and shells are backward-compatible. <a class="footnote-backref" href="#fnref:versions" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:limit">
<p>This post covers the higher-level functionality. You can fetch a precise list of implemented functionalities with <code>tezos-client --mode mockup rpc list</code>. <a class="footnote-backref" href="#fnref:limit" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:technical">
<p>Technical documentation describes this as: <em>mockup mode defaults to an unspecified protocol</em>. <a class="footnote-backref" href="#fnref:technical" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:updates">
<p>A Tezos blockchain has a self-amendment mechanism allowing to modify the protocol, subject to community votes. These self-amendments are called <strong>protocol updates</strong>. <a class="footnote-backref" href="#fnref:updates" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:immediate">
<p>Note that valid transfers in stateful mode are immediate, without any networking or consensus mechanisms. In contrast, a sandboxed Tezos blockchain maintains an (emulated) network and, most expensively, it adheres to the full consensus and baking mechanisms of a proper blockchain. <a class="footnote-backref" href="#fnref:immediate" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:irrelevant">
<p>In mockup mode, step 1 is irrelevant. This isn’t because there is no network — we might still simulate one locally. It’s because there’s no node. <a class="footnote-backref" href="#fnref:irrelevant" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:forging">
<p>Note the word ‘forge’ in the code above. Wait, we can explain: it’s a <a href="https://dictionary.cambridge.org/dictionary/english/term-of-art">term of art</a>! <em>Forging</em> or <em>minting</em> is <a href="https://hackernoon.com/what-is-proof-of-stake-8e0433018256">standard terminology in Proof-of-Stake systems</a> (like Tezos) for the operation of creating a block, and corresponds to ‘mining’ in Proof-of-Work systems (like Bitcoin). Thus, the <a href="https://en.wiktionary.org/wiki/forge#Etymology_1">etymology and meaning</a> of a phrase like “forgers forge forged transactions” is … entirely respectable. <a class="footnote-backref" href="#fnref:forging" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn:history">
<p>With the current Delphi protocol (<a href="https://research-development.nomadic-labs.com/delphi-official-release.html">announcement</a>, <a href="https://research-development.nomadic-labs.com/delphi-changelog.html">changelog</a>) it takes sixty blocks for a transaction to be removed from the mempool (this is governed by <a href="https://gitlab.com/tezos/tezos/-/blob/master/src/proto_007_PsDELPH1/lib_protocol/alpha_context.ml#L242">the <code>max_operations_ttl</code> parameter</a>). <a class="footnote-backref" href="#fnref:history" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
<li id="fn:internal">
<p>For instance, we used mockup mode to test the <a href="https://gitlab.com/metastatedev/tezos/-/merge_requests/399">bugfix</a> for <a href="https://tezos.gitlab.io/protocols/008_edo.html?highlight=tickets#comb-pairs">comb pairs</a> in the Delphi protocol. <br> We also used mockup mode to develop the <a href="https://gitlab.com/metastatedev/tezos/-/merge_requests/303/diffs?commit_id=9df901da350b20b4b2bd7125d7b7cddb80a957af">four smart contracts</a> (<code>ticket_builder_fungible.tz</code>, <code>ticket_builder_non_fungible.tz</code>, <code>ticket_wallet_fungible.tz</code> and <code>ticket_wallet_non_fungible.tz</code>) which are examples of using <a href="https://tezos.gitlab.io/protocols/008_edo.html?highlight=tickets#tickets">the new tickets feature</a> of Edo in a specific example implementation. Here, mockup mode tightened our developers’ development cycle in Michelson Emacs mode, which now uses mockup mode as a default engine to derive type information. <br> Before <a href="https://gitlab.com/tezos/tezos/-/commit/199a1e82bd6f6aa79b2db7705075d4742d0ddd28">commit 199a1e82</a> Emacs integration used sandboxed mode. This required defining a <code>tezos-client</code> alias for inside the Emacs mode, and launching a node, all before starting up Emacs. With mockup mode we just create a state and operate on it directly, for immediate feedback. <a class="footnote-backref" href="#fnref:internal" title="Jump back to footnote 9 in the text">↩</a></p>
</li>
</ol>
</div>Announcing the Edo Release!2020-11-30T12:00:00+01:002020-11-30T12:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2020-11-30:/announcing-the-edo-release.html<p>Summary:</p>
<p><em>This is a joint announcement from Nomadic Labs, Marigold, and Metastate.</em></p>
<p>A couple of weeks ago, we were proud to see the “Delphi” upgrade to the Tezos protocol <a href="https://www.prnewswire.com/news-releases/delphi-the-latest-tezos-upgrade-is-live-301172089.html">go live</a>. This week, we are proud to announce our latest protocol upgrade proposal, “Edo”. As usual, Edo’s true name …</p><p>Summary:</p>
<p><em>This is a joint announcement from Nomadic Labs, Marigold, and Metastate.</em></p>
<p>A couple of weeks ago, we were proud to see the “Delphi” upgrade to the Tezos protocol <a href="https://www.prnewswire.com/news-releases/delphi-the-latest-tezos-upgrade-is-live-301172089.html">go live</a>. This week, we are proud to announce our latest protocol upgrade proposal, “Edo”. As usual, Edo’s true name is its hash, which is <code>PtEdoTezd3RHSC31mpxxo1npxFjoWWcFgQtxapi51Z8TLu6v6Uq</code>.</p>
<p>Why is Edo being proposed when Delphi has only been in place for a short while? Although Delphi went live on November 12th, it was proposed on September 3rd. In the intervening months, we’ve been hard at work on the core Tezos software, and we’ve made significant improvements that we want to share with the users of the network. In particular, we have now completed a number of improvements that were in progress at the time that the interim Delphi update was proposed.</p>
<p>The Tezos protocol currently provides windows for new proposals every several months; one such window is now open. As we explain in <a href="https://forum.tezosagora.org/t/regular-scheduling-for-our-tezos-proposals/2331">this blog post</a>, we intend for the foreseeable future to take advantage of every such opportunity, proposing upgrades that incorporate the improvements that have been completed in the intervening months since the last proposal.</p>
<p>Most cryptocurrency networks cannot be updated on a regular basis; they have no mechanism that overcomes the high coordination costs associated with protocol changes. Tezos, however, possesses an on-chain self-governance mechanism, as well as a mechanism for self-amendment without forks, and so we can propose updates to the chain which, if adopted by its users, are then automatically implemented. We intend to take full advantage of that mechanism going forward to make Tezos better and better with every proposal.</p>
<p>As for Edo itself: a full list of the changes can be found on <a href="https://tezos.gitlab.io/protocols/008_edo.html">this documentation page</a>. In summary, however, the proposal contains some minor bug fixes, some additional improvements to performance and gas costs (albeit not as extreme as the ones in Delphi), the addition of a so-called “adoption period” to the voting schedule, and two important new features that we have been working on for some time: Sapling, and Tickets.</p>
<p>Sapling is a protocol originally developed by the Electric Coin Company for the Zcash project which implements shielded transactions. Our proposal allows smart contract developers to easily integrate Sapling in their smart contracts and create privacy-conscious applications. Because Tezos can be amended, it was possible for us to add this exciting new feature directly to Tezos itself.</p>
<p>Since our <a href="https://research-development.nomadic-labs.com/sapling-integration-in-tezos-tech-preview.html">initial announcement</a> of Sapling, the integration with Tezos has seen extensive testing and has been enhanced in numerous ways; we have also improved the performance.</p>
<p>Tickets are a convenient mechanism for smart contracts to grant portable permissions to other smart contracts or to issue tokens. While it’s possible to achieve this with existing programming patterns, tickets make it much easier for developers to write secure and composable contracts.</p>
<p>The “adoption period” (sometimes referred to as the “fifth period”) is an important improvement we have wanted to make to the governance mechanism for some time. Just like any other feature of the protocol, Tezos protocol amendments may make changes to the amendment process itself. Up until now, new versions of the protocol have gone live (that is, have been “activated”) one block after voting has been completed, which in practice is only sixty seconds. This has made it difficult for some Tezos bakers, indexers, and other users of the network to assure seamless upgrades of their nodes. We have also seen instances where the lack of certainty about whether an upgrade would be adopted caused some users to delay preparations until the last moment.</p>
<p>Under the new system, instead of four periods of eight cycles during voting, we propose to have five periods of five cycles. The new fifth period, the adoption period, will be a five cycle (approximately two weeks) gap between the adoption of the new protocol and the time when it is activated. This will aid in assuring seamless protocol transitions. (We anticipate some additional minor tweaks to the voting schedule may occur in coming protocol proposals.)</p>
<p>Some readers may notice that Baking Accounts, a feature that has been in the works for some time, is not included in Edo. Although the core Baking Accounts software is complete and reliable, we are not yet satisfied that the migration mechanisms needed to update the chain when Baking Accounts are activated are as seamless as we can make them. In the past, some migrations have caused delays to on-chain transactions occurring around the time of an upgrade, and our tests of the Baking Accounts migration indicate that it could take a considerable period of time. Going forward, as we will be upgrading the protocol quite frequently, we intend to minimize these migration times and any disruption to the network. We are thus working on optimizing our migration mechanisms.</p>
<p>Following our current policy of not slowing down deployment of completed features for ones that are not yet finished, we have held Baking Accounts back for the moment. We hope that Baking Accounts will be a feature of the next protocol proposal, which, if Edo is adopted, should occur in about three months.</p>Announcing Ebetanet, the Edo Preview Network!2020-11-20T15:00:00+01:002020-11-20T15:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2020-11-20:/announcing-ebetanet-the-edo-preview-network.html<p>We have just spawned a test network for a beta version of the Edo protocol, which we plan to propose as the next (008) Tezos protocol upgrade.</p>
<p><em>The code running on the test network is our release candidate for Edo. We anticipate that the beta period will last only one to two weeks before our proposal is final. Please participate by testing it now!</em></p>
<p>We plan to replace this test network with Edonet, the successor of Delphinet, once we get out of beta and formally propose Edo.</p>
<p>If you are interested in participating in this test network, you can checkout the <code>ebetanet-release</code> <a href="https://www.gitlab.com/tezos/tezos">branch</a> of the repository and build it from source:</p>
<div class="highlight"><pre><span></span><code>git fetch
git checkout ebetanet-release
make build-deps
eval $(opam env)
make
</code></pre></div>
<p>Note that you need to have installed the Rust compiler first. To install it, use the same <a href="http://tezos.gitlab.io/introduction/howtoget.html#environment">instructions</a> as those for the <code>master</code> branch.</p>
<p>Docker images are also available with tag <code>ebetanet-release</code>.</p>
<p>This branch is configured to join Ebetanet automatically so there is no need to run your node with the <code>--network ebetanet</code> command-line option.</p>
<p><strong>Important note:</strong> To avoid mistakes, this branch cannot join Mainnet as there is no <code>--network mainnet</code> option available.</p>
<p>We thank Smart Chain Arena for providing an initial <a href="https://ebetanet.smartpy.io">publicly available node</a>.</p>Cortez End of Support2020-11-16T15:00:00+01:002020-11-16T15:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2020-11-16:/cortez-end-of-support.html<p>In a context where Nomadic Labs aims to concentrate on its high value activities, we plan to refocus our efforts on projects and tools that are directly related to the heart of Tezos and its
economic protocol. As a result, Nomadic Labs decided to discontinue its support and maintenance of both the Android and iOS versions of the Tezos mobile wallet, Cortez.</p>
<p>After a grace period running from now to 15 February 2021, Nomadic Labs will no longer guarantee Cortez’s functionality, meaning that <strong>from 16 February 2021 we will not be responsible for keeping Cortez up-to-date with the Tezos blockchain,</strong> and users use Cortez at their own risk.</p>
<p>We recommend Cortez mobile app users export their private key(s) and switch to another Tezos wallet. Both <a href="https://magmawallet.io/">Magma</a> and <a href="https://cryptonomic.tech/galleon.html">Galleon</a> will allow you to import private keys from your mnemonic seed phrase. Alternatively, users can also make use of the Tezos <a href="https://tezos.gitlab.io/api/cli-commands.html">command line wallet</a>.</p>
<p><strong>How to export your private key(s) out of Cortez</strong></p>
<p>Cortez has an export function for you to extract the seed phrase for your accounts. This will allow you to access the same accounts/funds in other wallets. Note: store your seed phrase in a secure location!</p>
<p><strong><span class="caps">ANDROID</span></strong>:</p>
<ol>
<li>
<p>From the Dashboard, select the menu at the top right and choose “Key management”.</p>
<p><img alt="screenshot" src="images/CortezEoS/01n.jpg"></p>
</li>
<li>
<p>Select “Export your 24 words”.</p>
<p><img alt="screenshot" src="images/CortezEoS/02n.jpg"></p>
</li>
<li>
<p>Authenticate your password.</p>
<p><img alt="screenshot" src="images/CortezEoS/03n.jpg"></p>
</li>
<li>
<p>Your seed phrase will be displayed.</p>
<p><img alt="screenshot" src="images/CortezEoS/04n.jpg"></p>
</li>
</ol>
<p><strong>iOS</strong>:</p>
<ol>
<li>
<p>From the main menu, select Settings and
choose to Export the seed phrase.</p>
<p><img alt="screenshot" src="images/CortezEoS/iOS/01n.png"></p>
</li>
<li>
<p>Authenticate your password.</p>
<p><img alt="screenshot" src="images/CortezEoS/iOS/02n.png"></p>
</li>
<li>
<p>Your seed phrase will be displayed.</p>
<p><img alt="screenshot" src="images/CortezEoS/iOS/03n.png"></p>
</li>
</ol>
<p>Note: Nomadic Labs will never ask you for your seed phrase! Keep your 24-word seed safe! You can use it to <span class="caps">IMPORT</span> your funds to a new wallet. Alternatively, you can also transfer your <span class="caps">XTZ</span> out of Cortez into a new wallet that supports Tezos.</p>
<p>The three month maintenance period starts now and ends on 15 February 2021.</p>
<p><strong> Cortez is open-source so could become community-driven:</strong></p>
<ul>
<li><a href="https://gitlab.com/nomadic-labs/cortez-android">The Android Cortez repository</a></li>
<li><a href="https://gitlab.com/nomadic-labs/cortez-ios">The <span class="caps">IOS</span> Cortez repository</a></li>
</ul>Smarter contracts thanks to Delphi (part 1/2)2020-11-13T00:00:00+01:002020-11-13T00:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2020-11-13:/smarter-contracts-thanks-to-delphi-part-12.html<p><a href="https://research-development.nomadic-labs.com/delphi-official-release.html">Delphi</a> is the successor to the <a href="https://tezos.gitlab.io/protocols/006_carthage.html">Carthage protocol</a>.
Delphi’s main difference from Carthage is that gas costs are lower, so that smart contracts can compute more before hitting the Delphi/Carthage per-operation gas limit of 1,040,000 gas units (<strong>gu</strong>).</p>
<p>In this post we quantify the difference that Delphi’s lower gas costs will make:</p>
<ul>
<li>We start with a description and justification of the Michelson gas model; and then</li>
<li>we showcase the expected gains for some smart contracts chosen to illustrate the Delphi model’s advantages.</li>
</ul>
<p>Measuring the gas gains for real-world smart contracts will be the topic of another post.</p>
<h1 id="1-an-overview-of-the-michelson-gas-model"><a class="toclink" href="#1-an-overview-of-the-michelson-gas-model">1. An overview of the Michelson gas model</a></h1>
<p>To recap: gas limits guarantee that block validators only need a fraction of the time interval between blocks to validate a new block.</p>
<p>The exact fraction depends on the speed of the node’s hardware. Tezos tries to be inclusive, so we want even slow hardware to be able to sync the chain in real time.</p>
<p>As we wrote in our <a href="https://research-development.nomadic-labs.com/michelson-updates-in-005.html#gas-cost-overhaul">last gas post</a>:</p>
<blockquote>
<p><em>overestimating gas costs prevents developers from writing more interesting contracts, while underestimating these costs leads to possible attacks. It is important to get this right.</em></p>
</blockquote>
<p>In fact, there are two ways to allow more computation in a block:</p>
<ol>
<li>improve the performance of block validation and decrease the gas costs accordingly, or</li>
<li>refine the gas model when it overestimates gas costs.</li>
</ol>
<p>For the Delphi proposal, we did both.</p>
<h2 id="11-execution-steps-of-a-smart-contract-call"><a class="toclink" href="#11-execution-steps-of-a-smart-contract-call">1.1 Execution steps of a smart contract call</a></h2>
<p>A Tezos smart contract is a type of Tezos account consisting of:</p>
<ol>
<li>the <strong>smart contract balance</strong>,</li>
<li>the <strong>smart contract code</strong>, which is executable code in the Michelson programming language, and</li>
<li>the <strong>smart contract storage</strong>, used to save the contract’s state between calls.</li>
</ol>
<p>A transfer calls a smart contract by invoking it along with a <strong>parameter</strong>. The Tezos protocol then executes<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">1</a></sup> the smart contract code, passing it two inputs:</p>
<ol>
<li>the parameter, and</li>
<li>the contract’s current storage.</li>
</ol>
<p>Outputs after execution are:</p>
<ul>
<li>updated storage, and</li>
<li>a (possibly empty) list of operations to be added to the queue of pending operations.<sup id="fnref:3"><a class="footnote-ref" href="#fn:3">2</a></sup></li>
</ul>
<p>The Tezos protocol uses three distinct formats to represent Michelson scripts and values, listed here in increasingly higher levels of abstraction:</p>
<ol>
<li><em>Raw byte sequences:</em> is how data is stored on disk, or in operation payloads (in particular, parameters are stored as raw byte sequences).</li>
<li><em><a href="https://tezos.gitlab.io/whitedoc/micheline.html">Micheline</a>:</em> the Micheline format is protocol-independent (think: <span class="caps">XML</span> or Json for Michelson code) and portable across protocol versions. Micheline data has a tree-like structure so it is easier to manipulate than byte sequences; but it is still untyped.</li>
<li><em>The Michelson internal representation:</em> this format is specific to each version of the Tezos protocol. Michelson expressions written in this format are guaranteed to be well-typed. It is the only format known to the Michelson interpreter.</li>
</ol>
<p><a name="calling-a-contract"></a> When a Tezos smart contract is called — either due to some initial call to it, or as a consequence of working through a queue of accumulated operations e.g. from step 8 below — the Tezos protocol performs the following steps:</p>
<ol>
<li><em>Read the contract’s script and storage</em> from disk, as raw byte sequences (the parameter is already in memory).</li>
<li><em><strong>Deserialise</strong> (convert) the script, storage, and parameter</em> to <a href="https://tezos.gitlab.io/whitedoc/micheline.html">Micheline</a>.</li>
<li><em>Convert Micheline to the internal representation.</em> The protocol code calls this step <strong>parsing</strong>; the command-line client calls it <strong>type-checking</strong> (since type errors might be thrown).</li>
<li><em>Interpret (execute) the smart contract</em> using as inputs its storage and the parameter. At the end of this step, we obtain the updated storage, and a list of new operations.</li>
<li><em><strong>Unparse</strong> updated storage</em> — convert from the internal representation to Micheline.</li>
<li><em><strong>Serialise</strong> this Micheline expression</em> — convert it to a byte sequence.</li>
<li><em>Write the byte sequence</em> to disk.</li>
<li><em>Queue the list of new operations</em> to apply.</li>
<li><em>Loop to 1</em>, until the operation queue is empty.</li>
</ol>
<h2 id="12-limiting-execution-with-gas"><a class="toclink" href="#12-limiting-execution-with-gas">1.2 Limiting execution, with gas</a></h2>
<p>Each of the execution steps above could take arbitrary time, so to guarantee that nodes can check the validity of operations in reasonable time, the protocol imposes a hard limit on computation and considers as invalid any operations that exceed it.
This limit cannot just be a timeout: then fast nodes would have different valid operations than slower ones — and the network would not reach consensus.</p>
<p>Instead, we rely on <strong>gas</strong>; a hardware-independent abstraction of the time needed to validate operations that is built in to the Tezos protocol.</p>
<p>Here are the usual reasons we consume gas (relative weights depend on the smart contract being executed):</p>
<ol>
<li>reading from disk,</li>
<li>deserialisation,</li>
<li>parsing,</li>
<li>interpretation, including a fixed per-operation gas cost, </li>
<li>unparsing,</li>
<li>serialisation, and</li>
<li>writing to disk.</li>
</ol>
<p>Gas prevents overlong operations, so the chain’s security depends on good estimates of how long each task above will take:</p>
<ul>
<li><em>An underestimate</em> is a security risk because it makes it cheap to launch a denial-of-service attack on a node, using a smart contract.</li>
<li><em>An overestimate</em> will unduly restrict the complexity of the smart contracts that can be run, which may annoy users but is not a security risk.</li>
</ul>
<p>Thus, when we <a href="https://tezos.foundation/tezos-betanet-launch/">launched the Betanet</a>, our gas constraints were deliberately overestimated, intended to be decreased later by protocol amendments.</p>
<h2 id="13-estimating-gas-costs-and-getting-it-right"><a class="toclink" href="#13-estimating-gas-costs-and-getting-it-right">1.3 Estimating gas costs (and getting it right)</a></h2>
<p>But how far can we safely decrease gas costs in practice? To quantify this, we performed two kinds of benchmarks: </p>
<ul>
<li><strong>Macro benchmarks</strong> fill a block with operations designed to stress some particular source of gas consumption (e.g. computation, or disk accesses).</li>
<li><strong>Micro benchmarks</strong> measure the time it takes to execute functions of the protocol on random inputs (of various sizes). This helps build a predictive model of the time taken by the function, so that the gas cost can precisely reflect it.</li>
</ul>
<p>In the previous Athens and Babylon amendments we adjusted the gas model:</p>
<p>For <a href="https://research-development.nomadic-labs.com/athens-our-proposals-for-the-first-voted-amendment.html#gas-limits">the Athens proposals</a>, we did macro benchmarks to refine the relative costs of disk accesses and computations. These benchmarks showed that disk accesses were too expensive compared to the other costs and we divided the relative cost of disk accesses by 2 to reflect this.</p>
<p>For the Babylon proposal, we micro-benchmarked the interpretation time of most instructions on random data of various sizes and interpolated cost models to all the benchmarked instructions. This led to a <a href="https://research-development.nomadic-labs.com/michelson-updates-in-005.html#gas-cost-overhaul">significant decrease of the interpretation costs of most instructions</a>.</p>
<p>For the current Delphi proposal:</p>
<ul>
<li>We developed a generator of well-typed Michelson code of arbitrary size and used it to micro-benchmark serialisation and parsing functions.</li>
<li>We optimised the parsing functions. </li>
<li>We optimised the Michelson interpreter.</li>
<li>We updated the Babylon instruction-per-instruction benchmarks to reflect the new optimisations of the interpreter and reduced the interpretation costs accordingly.</li>
<li>We benchmarked disk accesses on modern hardware and revised the corresponding gas model.</li>
</ul>
<p>The modifications of the gas model brought by the Delphi proposal are documented in the following merge requests:</p>
<ul>
<li><a href="https://gitlab.com/metastatedev/tezos/-/merge_requests/282">on the Michelson interpreter</a>,</li>
<li><a href="https://gitlab.com/metastatedev/tezos/-/merge_requests/283">on disk I/Os</a>,</li>
<li><a href="https://gitlab.com/metastatedev/tezos/-/merge_requests/285">on parsing</a>, and</li>
<li><a href="https://gitlab.com/metastatedev/tezos/-/merge_requests/287">a final cleanup</a>.</li>
</ul>
<p>Now, we will <em>measure</em> the changes in gas consumption for contracts that are intensive in one of the components and not the others.</p>
<h1 id="2-theoretical-limits"><a class="toclink" href="#2-theoretical-limits">2. Theoretical limits</a></h1>
<p>The following Michelson scripts are not realistic; they are designed to measure the gas gains for the Delphi proposal:</p>
<p><a name="io"></a></p>
<h2 id="21-gas-costs-for-disk-access-read-write"><a class="toclink" href="#21-gas-costs-for-disk-access-read-write">2.1 Gas costs for disk access (read / write)</a></h2>
<p>To avoid deserialisation and parsing costs, we read and write lots of data of type <code>bytes</code>.
Before reaching the gas limit per operation, we hit another limit defined by the protocol: the <strong>storage increase limit</strong>.</p>
<p>A single transaction may increase the contract storage by no more than 60kB (1kB = 1000 bytes) — we can build larger byte sequences dynamically; see later.
To write more than 60kB in a single transaction we must also remove data in the same transaction.</p>
<p>Here is our Michelson script for a smart contract to stress the disk limits:</p>
<div class="highlight"><pre><span></span><code><span class="err">#</span><span class="w"> </span><span class="n">Script</span><span class="w"> </span><span class="n">io</span><span class="p">.</span><span class="n">tz</span>
<span class="k">parameter</span><span class="w"> </span><span class="p">(</span><span class="ow">or</span><span class="w"> </span><span class="p">(</span><span class="n">unit</span><span class="w"> </span><span class="o">%</span><span class="n">add32K</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="ow">or</span><span class="w"> </span><span class="p">(</span><span class="n">nat</span><span class="w"> </span><span class="o">%</span><span class="k">write</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">unit</span><span class="w"> </span><span class="o">%</span><span class="k">read</span><span class="p">)));</span>
<span class="n">storage</span><span class="w"> </span><span class="p">(</span><span class="n">pair</span><span class="w"> </span><span class="p">(</span><span class="n">nat</span><span class="w"> </span><span class="o">%</span><span class="n">counter</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="n">big_map</span><span class="w"> </span><span class="o">%</span><span class="n">tank</span><span class="w"> </span><span class="n">nat</span><span class="w"> </span><span class="n">bytes</span><span class="p">));</span>
<span class="n">code</span>
<span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">UNPAPAIR</span><span class="p">;</span>
<span class="w"> </span><span class="n">IF_LEFT</span>
<span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="o">%</span><span class="n">add32K</span><span class="p">(</span><span class="n">_</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="n">unit</span><span class="p">)</span><span class="err">:</span><span class="w"> </span><span class="k">add</span><span class="w"> </span><span class="mi">32</span><span class="w"> </span><span class="n">KiB</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">storage</span>
<span class="w"> </span><span class="k">DROP</span><span class="p">;</span>
<span class="w"> </span><span class="n">PUSH</span><span class="w"> </span><span class="n">nat</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"> </span><span class="k">ADD</span><span class="p">;</span>
<span class="w"> </span><span class="n">PUSH</span><span class="w"> </span><span class="n">bytes</span><span class="w"> </span><span class="mh">0x00</span><span class="p">;</span>
<span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="mi">15</span><span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="ow">SOME</span><span class="p">;</span><span class="w"> </span><span class="n">SWAP</span><span class="p">;</span><span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="n">DUG</span><span class="w"> </span><span class="mi">3</span><span class="p">;</span><span class="w"> </span><span class="k">UPDATE</span><span class="p">;</span><span class="w"> </span><span class="n">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="n">PAIR</span><span class="p">;</span>
<span class="w"> </span><span class="err">}</span>
<span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">IF_LEFT</span>
<span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="o">%</span><span class="k">write</span><span class="p">(</span><span class="n">n</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="n">nat</span><span class="p">)</span><span class="err">:</span><span class="w"> </span><span class="n">store</span><span class="w"> </span><span class="mi">2</span><span class="o">^</span><span class="n">n</span><span class="w"> </span><span class="n">bytes</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">big_map</span>
<span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="k">First</span><span class="w"> </span><span class="n">clear</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">big_map</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">decrease</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">storage</span><span class="w"> </span><span class="k">size</span><span class="w"> </span><span class="n">diff</span>
<span class="w"> </span><span class="n">DUG</span><span class="w"> </span><span class="mi">2</span><span class="p">;</span>
<span class="w"> </span><span class="n">RENAME</span><span class="w"> </span><span class="nv">@counter</span><span class="p">;</span>
<span class="w"> </span><span class="n">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="n">PUSH</span><span class="w"> </span><span class="nv">@index</span><span class="w"> </span><span class="n">nat</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="n">PUSH</span><span class="w"> </span><span class="n">bool</span><span class="w"> </span><span class="k">True</span><span class="p">;</span>
<span class="w"> </span><span class="n">LOOP</span>
<span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="n">DIP</span><span class="w"> </span><span class="err">{</span><span class="w"> </span><span class="k">NONE</span><span class="w"> </span><span class="n">bytes</span><span class="p">;</span><span class="w"> </span><span class="n">SWAP</span><span class="p">;</span><span class="w"> </span><span class="k">UPDATE</span><span class="w"> </span><span class="err">}</span><span class="p">;</span>
<span class="w"> </span><span class="n">PUSH</span><span class="w"> </span><span class="n">nat</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"> </span><span class="k">ADD</span><span class="w"> </span><span class="nv">@index</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="w"> </span><span class="nv">@index</span><span class="p">;</span><span class="w"> </span><span class="n">DUP</span><span class="w"> </span><span class="nv">@counter</span><span class="w"> </span><span class="mi">4</span><span class="p">;</span><span class="w"> </span><span class="n">CMPGE</span>
<span class="w"> </span><span class="err">}</span><span class="p">;</span>
<span class="w"> </span><span class="k">DROP</span><span class="p">;</span><span class="w"> </span><span class="n">SWAP</span><span class="p">;</span><span class="w"> </span><span class="k">DROP</span><span class="p">;</span><span class="w"> </span><span class="n">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="n">do</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span>
<span class="w"> </span><span class="n">DIP</span><span class="w"> </span><span class="err">{</span><span class="w"> </span><span class="n">PUSH</span><span class="w"> </span><span class="n">bytes</span><span class="w"> </span><span class="mh">0x00</span><span class="w"> </span><span class="err">}</span><span class="p">;</span>
<span class="w"> </span><span class="nc">INT</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="n">GT</span><span class="p">;</span>
<span class="w"> </span><span class="n">LOOP</span>
<span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">PUSH</span><span class="w"> </span><span class="n">nat</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"> </span><span class="n">SWAP</span><span class="p">;</span><span class="w"> </span><span class="n">SUB</span><span class="p">;</span>
<span class="w"> </span><span class="n">DIP</span><span class="w"> </span><span class="err">{</span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="nf">CONCAT</span><span class="err">}</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span><span class="w"> </span><span class="n">GT</span>
<span class="w"> </span><span class="err">}</span><span class="p">;</span>
<span class="w"> </span><span class="k">DROP</span><span class="p">;</span>
<span class="w"> </span><span class="ow">SOME</span><span class="p">;</span>
<span class="w"> </span><span class="n">PUSH</span><span class="w"> </span><span class="n">nat</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="k">UPDATE</span>
<span class="w"> </span><span class="err">}</span>
<span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="err">#</span><span class="w"> </span><span class="k">read</span><span class="p">(</span><span class="n">_</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="n">unit</span><span class="p">)</span><span class="err">:</span><span class="w"> </span><span class="k">read</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">bytes</span><span class="w"> </span><span class="n">that</span><span class="w"> </span><span class="n">have</span><span class="w"> </span><span class="n">been</span><span class="w"> </span><span class="n">stored</span><span class="w"> </span><span class="k">by</span><span class="w"> </span><span class="o">%</span><span class="k">write</span>
<span class="w"> </span><span class="k">DROP</span><span class="w"> </span><span class="mi">2</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="n">PUSH</span><span class="w"> </span><span class="n">nat</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span>
<span class="w"> </span><span class="k">GET</span><span class="p">;</span><span class="w"> </span><span class="k">DROP</span>
<span class="w"> </span><span class="err">}</span><span class="p">;</span>
<span class="w"> </span><span class="n">PUSH</span><span class="w"> </span><span class="n">nat</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"> </span><span class="n">PAIR</span>
<span class="w"> </span><span class="err">}</span><span class="p">;</span>
<span class="w"> </span><span class="n">NIL</span><span class="w"> </span><span class="k">operation</span><span class="p">;</span>
<span class="w"> </span><span class="n">PAIR</span>
<span class="w"> </span><span class="err">}</span>
</code></pre></div>
<p>The script <code>io.tz</code> above declares a storage of type <code>pair (nat %counter) (big_map %tank nat bytes)</code>. The <code>tank</code> big map serves as a data tank which we will fill in several transactions, and then empty all at once to get a lot of writing rights. <code>tank</code> is expected to contain at index <code>i</code> 32KiB (1KiB = 1024 bytes) of zeros if <code>i</code> is smaller than the counter <code>counter</code>, and nothing otherwise. This script features three entrypoints:</p>
<ul>
<li><code>add32K</code> inserts data into the tank. <br>
It adds 32KiB of data (note that 64KiB would exceed the storage size diff limit) in the first empty slot in the big map, and increments the counter to maintain the invariant.</li>
<li><code>write n</code> empties the tank. <br>
It puts <code>None</code> in all big map entries from <span class="math">\(0\)</span> to the value of the counter, resets the counter to <span class="math">\(0\)</span>, and puts a byte sequence of size <span class="math">\(2^n\)</span> at the first position in the big map.</li>
<li><code>read</code> accesses the first position in the big map and does nothing with it. <br>
This entrypoint returns the storage unchanged.</li>
</ul>
<p>We use the script <code>io.tz</code> as follows:</p>
<ol>
<li>We originate it on a default storage <code>Pair 0 {}</code>; and</li>
<li>we repeatedly call the <code>add32K</code> entrypoint to insert enough data in the tank; and</li>
<li>once the storage is big enough (experimentally, 64 calls to <code>add32K</code> are enough), we call <code>write</code> with the largest parameter that the gas limit allows; and finally</li>
<li>we call <code>read</code>.</li>
</ol>
<p>For the <code>read</code> entrypoint, thanks to the <code>--trace-stack</code> option of the client, we can compute the gas consumption of the <code>GET</code> instruction alone, so the gas consumption caused by reading the disk is well-isolated from other gas consumptions.</p>
<ul>
<li>In Carthage, the largest <code>n</code> we can pass the <code>write</code> entrypoint is <code>n=15</code>, which hence writes 32KiB (<span class="math">\(2^{15}\)</span>B) and consumes 535741 gu (thus slightly more than half the per-operation gas limit of 1,040,000 gu). Reading back these 32KiB costs approximately a third of the gas limit (the gas consumption of the <code>GET</code> instruction is 344481 gu).</li>
<li>In Delphi, we can reach <code>n=21</code> which writes 2MiB (<span class="math">\(2^{21}\)</span>B) and also consumes exactly 555330.160 gu (again slightly more than half the gas limit, which is the same as Carthage’s); the <code>GET</code> instruction of the <code>read</code> entrypoint then consumes almost as much gas (520045.437 gu).</li>
</ul>
<p>In summary, comparing the gas costs in Carthage and Delphi:</p>
<ul>
<li><strong>Writing large pieces of data costs about 62 times less</strong> (a 98.4% saving).</li>
<li><strong>Reading data costs about 42 times less</strong> (a 97.6% saving).</li>
</ul>
<p>Quantifying the real-world impact of these cost reductions on practical Delphi usage will be the topic of the future blog post — but we can note here that this is clearly a significant reduction.</p>
<p><a name="parsing"></a></p>
<h2 id="22-gas-costs-for-parsing-code-and-data"><a class="toclink" href="#22-gas-costs-for-parsing-code-and-data">2.2 Gas costs for parsing code and data</a></h2>
<p>Parsing costs are easy to decorrelate from the other sources of gas consumption thanks to the <code>tezos-client typecheck data</code> and <code>tezos-client typecheck script</code> commands that report exactly the amount of gas consumed at parsing. The Delphi gas model for parsing costs typically 40 times less than Carthage’s one.</p>
<p>To assess typechecking costs, we construct <a href="files/smarter-contracts-thanks-to-delphi-1/big_script.tz.gz">a synthetic script <code>big_script.tz</code></a>, which contains 4000 <code>DUP</code> and 4000 <code>DROP</code>.
We then typecheck this script via the <code>tezos-client typecheck script</code> command.</p>
<p>This costs 248168 gas units under Carthage but only 6003.655 gas units under Delphi; so in this example the <strong>Dephi gas model costs 41.37 times less</strong>.</p>
<p><a name="contract-interpretation"></a>
<a name="improved-gas-accounting"></a></p>
<h2 id="23-gas-costs-for-contract-interpretation"><a class="toclink" href="#23-gas-costs-for-contract-interpretation">2.3 Gas costs for contract interpretation</a></h2>
<p>Gas costs in the Michelson interpreter include</p>
<ul>
<li>costs for instruction interpretation as discussed — and <em>also</em></li>
<li>overhead costs in the Michelson interpreter due to the gas accounting system itself.</li>
</ul>
<p>In Delphi we have optimised both.
To evaluate the gains due to the Michelson interpreter optimisations in Delphi, we consider the scripts of two contracts:</p>
<ul>
<li><a href="#big_list.tz">constructing a large data structure (<code>big_list.tz</code>)</a>, and</li>
<li><a href="#arith.tz">computing a factorial function (<code>arith.tz</code>)</a>.</li>
</ul>
<h3 id="231-gas-costs-for-constructing-a-large-data-structure"><a class="toclink" href="#231-gas-costs-for-constructing-a-large-data-structure">2.3.1 Gas costs for constructing a large data structure</a></h3>
<p>Let’s first consider a contract that builds a huge list and does nothing with it:</p>
<p><a name="big_list.tz"></a></p>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span><span class="nv">Script</span><span class="w"> </span><span class="nv">big_list</span>.<span class="nv">tz</span>
<span class="nv">parameter</span><span class="w"> </span><span class="nv">nat</span><span class="c1">;</span>
<span class="nv">storage</span><span class="w"> </span><span class="nv">unit</span><span class="c1">;</span>
<span class="nv">code</span>
<span class="w"> </span>{
<span class="w"> </span><span class="nv">CAR</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">INT</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">NIL</span><span class="w"> </span><span class="nv">unit</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">UNIT</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">CONS</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">SWAP</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">DUP</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">GT</span><span class="c1">;</span>
<span class="w"> </span><span class="k">LOOP</span>
<span class="w"> </span>{
<span class="w"> </span><span class="nv">PUSH</span><span class="w"> </span><span class="nv">nat</span><span class="w"> </span><span class="mi">1</span><span class="c1">; SWAP; SUB;</span>
<span class="w"> </span><span class="nv">SWAP</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">DUP</span><span class="c1">; ITER {CONS};</span>
<span class="w"> </span><span class="nv">SWAP</span><span class="c1">; DUP; GT</span>
<span class="w"> </span>}<span class="c1">;</span>
<span class="w"> </span><span class="nv">DROP</span><span class="w"> </span><span class="mi">2</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">UNIT</span><span class="c1">; NIL operation; PAIR</span>
<span class="w"> </span>}
</code></pre></div>
<p>When the contract <code>big_list.tz</code> receives <code>n</code> as parameter, it:</p>
<ol>
<li>Builds a <code>list unit</code> that initially contains a single element.</li>
<li>Iterates <code>n</code> times in a loop that concatenates the list with itself (<code>DUP; ITER{CONS}</code>).</li>
<li>Empties the stack (<code>DROP 2</code>).</li>
</ol>
<p>So this contract pays gas for building a list of length <code>2^n</code>, and only for that.</p>
<ul>
<li>In Carthage we can set at most <code>n=20</code> and this consumes 996143 gas units.</li>
<li>In Delphi, the same contract with <code>n=20</code> consumes 230406 gas units.</li>
</ul>
<p>Thus, <strong>the cost in Delphi is 4.32 times less</strong>.</p>
<h3 id="232-gas-costs-for-arithmetic"><a class="toclink" href="#232-gas-costs-for-arithmetic">2.3.2 Gas costs for arithmetic</a></h3>
<p>Arithmetic in Michelson is implemented using the <a href="https://github.com/ocaml/Zarith">Zarith arbitrary-precision library</a>, a well-optimised OCaml wrapper of the <span class="caps">GMP</span> C library. Neither Zarith nor <span class="caps">GMP</span> has been modified in the development of Delphi and the cost models for arithmetic functions have been quite precise since Babylon, so there’s little hope for further optimisation in Delphi, for smart contracts that are mostly arithmetic computation.</p>
<p>To measure this, we wrote a simple loop-based factorial in Michelson:</p>
<p><a name="arith.tz"></a></p>
<div class="highlight"><pre><span></span><code><span class="err">#</span><span class="w"> </span><span class="n">Script</span><span class="w"> </span><span class="n">arith</span><span class="p">.</span><span class="n">tz</span>
<span class="k">parameter</span><span class="w"> </span><span class="n">nat</span><span class="p">;</span>
<span class="n">storage</span><span class="w"> </span><span class="n">unit</span><span class="p">;</span>
<span class="n">code</span>
<span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">CAR</span><span class="p">;</span>
<span class="w"> </span><span class="n">PUSH</span><span class="w"> </span><span class="nv">@acc</span><span class="w"> </span><span class="n">nat</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span>
<span class="w"> </span><span class="n">PUSH</span><span class="w"> </span><span class="nv">@i</span><span class="w"> </span><span class="n">nat</span><span class="w"> </span><span class="mi">2</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="w"> </span><span class="mi">4</span><span class="p">;</span>
<span class="w"> </span><span class="n">CMPGE</span><span class="p">;</span>
<span class="w"> </span><span class="n">LOOP</span>
<span class="w"> </span><span class="err">{</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="n">DIP</span><span class="w"> </span><span class="err">{</span><span class="n">MUL</span><span class="w"> </span><span class="nv">@acc</span><span class="err">}</span><span class="p">;</span>
<span class="w"> </span><span class="n">PUSH</span><span class="w"> </span><span class="n">nat</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"> </span><span class="k">ADD</span><span class="w"> </span><span class="nv">@i</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="n">DUP</span><span class="w"> </span><span class="mi">4</span><span class="p">;</span>
<span class="w"> </span><span class="n">CMPGE</span><span class="p">;</span>
<span class="w"> </span><span class="err">}</span><span class="p">;</span>
<span class="w"> </span><span class="k">DROP</span><span class="w"> </span><span class="mi">3</span><span class="p">;</span><span class="w"> </span><span class="n">UNIT</span><span class="p">;</span>
<span class="w"> </span><span class="n">NIL</span><span class="w"> </span><span class="k">operation</span><span class="p">;</span><span class="w"> </span><span class="n">PAIR</span>
<span class="w"> </span><span class="err">}</span>
</code></pre></div>
<p>As in the <a href="#big_list.tz"><code>big_list.tz</code> example</a>, the result of the computation is not stored, to isolate the cost of the computation from the cost of storing the result.</p>
<p>The biggest parameter that can be sent to this contract within the gas limit in Carthage is exactly 6400. With this parameter,</p>
<ul>
<li>1039686 gas units are consumed in Carthage and</li>
<li>389054 gas units are consumed in Delphi.</li>
</ul>
<p>Thus, <strong>the factorial script costs 2.67 times less in Delphi</strong>.</p>
<p><a name="inter-contract-calls"></a></p>
<h2 id="24-gas-costs-for-inter-contract-calls-in-two-parts"><a class="toclink" href="#24-gas-costs-for-inter-contract-calls-in-two-parts">2.4 Gas costs for inter-contract calls, in two parts</a></h2>
<h3 id="part-1-of-2-recursion"><a class="toclink" href="#part-1-of-2-recursion">Part 1 of 2: Recursion</a></h3>
<p>In Carthage, applying an operation has a minimal cost of 10 kgu (1 kgu = 1000 gu). In Delphi, the minimal cost is 1 kgu, thus ten times smaller. As we shall see below when we look at the <code>self_recursion.tz</code> script, this minimal cost is no longer a dominating cost for contracts that perform a lot of calls.</p>
<p>A simple way to test how many contract calls can be achieved in Carthage and in Delphi is to write a recursive contract that calls itself <code>n</code> times and then stops: it takes a number <code>n</code> as parameter and stops if <code>n</code> is <code>0</code>, and calls itself on <code>n-1</code> otherwise.</p>
<p>Here is the corresponding Michelson script:</p>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span><span class="nv">Script</span><span class="w"> </span><span class="nv">self_recusion</span>.<span class="nv">tz</span>
<span class="nv">parameter</span><span class="w"> </span><span class="nv">int</span><span class="c1">;</span>
<span class="nv">storage</span><span class="w"> </span><span class="nv">unit</span><span class="c1">;</span>
<span class="nv">code</span>
<span class="w"> </span>{
<span class="w"> </span><span class="nv">UNPAIR</span><span class="c1">; DUP; EQ;</span>
<span class="w"> </span><span class="k">IF</span>
<span class="w"> </span>{<span class="w"> </span><span class="nv">DROP</span><span class="c1">; NIL operation; PAIR }</span>
<span class="w"> </span>{
<span class="w"> </span><span class="nv">PUSH</span><span class="w"> </span><span class="nv">int</span><span class="w"> </span><span class="mi">1</span><span class="c1">; SWAP; SUB;</span>
<span class="w"> </span><span class="nv">SELF</span><span class="c1">; PUSH mutez 0; DIG 2; TRANSFER_TOKENS;</span>
<span class="w"> </span><span class="nv">NIL</span><span class="w"> </span><span class="nv">operation</span><span class="c1">; SWAP; CONS; PAIR</span>
<span class="w"> </span>}
<span class="w"> </span>}
</code></pre></div>
<ul>
<li>
<p>In Carthage, we measured that each recursive call of <code>self_recursion.z</code> consumes 13178 gas units, so the largest <code>n</code> we can send to this contract within the protocol-set per-operation gas limit (1,040,000 gu) is <code>n=77</code>.</p>
</li>
<li>
<p>In Delphi, each recursive call of this contract only consumes 2871.559 gas, so the largest <code>n</code> we can send to this contract within the per-operation gas limit (identical to Carthage’s) is <code>n=361</code>.</p>
</li>
</ul>
<p>We see that in Carthage the gas costs of <code>self_recursion.tz</code> are dominated by the 10 kgu cost of operation application, which represents about 3/4th of its total gas consumption. In Delphi, it only represents about 1/3rd of the total.</p>
<h3 id="part-2-of-2-a-chain-of-calls"><a class="toclink" href="#part-2-of-2-a-chain-of-calls">Part 2 of 2: A chain of calls</a></h3>
<p>Calling a contract (recursively or otherwise) incurs a non-trivial gas cost, corresponding to loading, deserialising, and type-checking the contract as well as checking that the parameter to the call has the right type (cf. <a href="#calling-a-contract">discussion above</a>).</p>
<p>Since typical inter-contract interactions are not recursive, we concentrate here on non-recursive calls.</p>
<p>To test non-recursive contract calls, we build a chain of smart contracts. Each contract calls the next until we reach the last contract in the chain, which then originates a new contract to grow the chain. More precisely:</p>
<ul>
<li>The contracts in the chain have two possible states called <code>forwarding</code> and <code>final</code>.</li>
<li>In <code>forwarding</code> state, the contract<ul>
<li>stores an address,</li>
<li>casts it to <code>contract unit</code> (the type of contracts expecting no parameter), and</li>
<li>calls it.</li>
</ul>
</li>
<li>In <code>final</code> state, the contract<ul>
<li>originates a copy of itself with an initial storage in <code>final</code> state,</li>
<li>stores the address of the newly-originated contract, and</li>
<li>switches to <code>forwarding</code> mode.</li>
</ul>
</li>
</ul>
<p>Unfortunately, there is no instruction in Michelson to originate a copy of oneself; the only way to originate a contract from Michelson is with the <code>CREATE_CONTACT</code> instruction, which expects as <em>static</em> argument the full script of the contract to originate. Therefore, to originate a copy of the current contract, we cannot use <code>CREATE_CONTRACT</code> directly in the contract script: there is no finite script satisfying the equation <code>script = { ...; CREATE_CONTRACT script ...; ... }</code>. We need to untie this recursive<sup id="fnref:4"><a class="footnote-ref" href="#fn:4">3</a></sup> equation and solve it at run-time, by either placing part of the script in the storage (using a lambda) or in the script of another contract.</p>
<p>We choose the former option (to put it in the storage) and leave the motivated reader to implement a solution based on the latter as an exercise.</p>
<p>The lambda that the contract stores needs to return both the origination operation and the address of the newly created contract, so its return type is <code>pair address operation</code>. As input, it needs some data to initialise the storage of the freshly-originated contract, which is basically the lambda itself in serialised form.</p>
<p>The trick to solve the recursive equation mentioned above is to have the lambda deserialise its own code using the <code>UNPACK</code> instruction — so the input type of the lambda is <code>bytes</code> and its code starts with <code>UNPACK (lambda bytes (pair address operation))</code> — followed by code to originate the contract using the <code>CREATE_CONTRACT</code> instruction.</p>
<p>The complete script of the contract used by each link in the call chain is:</p>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span><span class="nv">Script</span><span class="w"> </span><span class="nv">link_contract</span>.<span class="nv">tz</span>
<span class="nv">parameter</span><span class="w"> </span><span class="nv">unit</span><span class="c1">;</span>
<span class="nv">storage</span><span class="w"> </span><span class="ss">(</span><span class="nv">or</span><span class="w"> </span><span class="ss">(</span><span class="nv">lambda</span><span class="w"> </span><span class="o">%</span><span class="nv">final</span><span class="w"> </span><span class="nv">bytes</span><span class="w"> </span><span class="ss">(</span><span class="nv">pair</span><span class="w"> </span><span class="nv">address</span><span class="w"> </span><span class="nv">operation</span><span class="ss">))</span><span class="w"> </span><span class="ss">(</span><span class="nv">address</span><span class="w"> </span><span class="o">%</span><span class="nv">forward</span><span class="ss">))</span><span class="c1">;</span>
<span class="nv">code</span>
<span class="w"> </span>{<span class="w"> </span><span class="nv">NIL</span><span class="w"> </span><span class="nv">operation</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">SWAP</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">CDR</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">IF_LEFT</span>
<span class="w"> </span>{
<span class="w"> </span>#<span class="w"> </span><span class="nv">in</span><span class="w"> </span><span class="o">%</span><span class="nv">final</span><span class="w"> </span><span class="nv">mode</span>,<span class="w"> </span><span class="k">call</span><span class="w"> </span><span class="nl">the</span><span class="w"> </span><span class="nv">stored</span><span class="w"> </span><span class="nv">lambda</span><span class="w"> </span><span class="nv">on</span><span class="w"> </span><span class="nv">itself</span>
<span class="w"> </span><span class="nv">DUP</span><span class="w"> </span><span class="c1">;</span>
<span class="w"> </span><span class="nv">PACK</span><span class="c1">;</span>
<span class="w"> </span><span class="k">EXEC</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">UNPAIR</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">DIP</span>{<span class="nv">CONS</span>}
<span class="w"> </span>}
<span class="w"> </span>{
<span class="w"> </span>#<span class="w"> </span><span class="nv">in</span><span class="w"> </span><span class="o">%</span><span class="nv">forward</span><span class="w"> </span><span class="nv">mode</span>,<span class="w"> </span><span class="k">call</span><span class="w"> </span><span class="nl">the</span><span class="w"> </span><span class="nv">stored</span><span class="w"> </span><span class="nv">address</span>
<span class="w"> </span><span class="nv">DUP</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">CONTRACT</span><span class="w"> </span><span class="nv">unit</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">ASSERT_SOME</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">PUSH</span><span class="w"> </span><span class="nv">mutez</span><span class="w"> </span><span class="mi">0</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">UNIT</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">TRANSFER_TOKENS</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">SWAP</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">DIP</span>{<span class="nv">CONS</span>}
<span class="w"> </span>}<span class="c1">;</span>
<span class="w"> </span>#<span class="w"> </span><span class="nv">in</span><span class="w"> </span><span class="nv">both</span><span class="w"> </span><span class="nv">cases</span>,<span class="w"> </span><span class="nv">we</span><span class="w"> </span><span class="k">end</span><span class="w"> </span><span class="nv">in</span><span class="w"> </span><span class="o">%</span><span class="nv">forward</span><span class="w"> </span><span class="nv">mode</span>
<span class="w"> </span><span class="nv">RIGHT</span><span class="w"> </span><span class="ss">(</span><span class="nv">lambda</span><span class="w"> </span><span class="nv">bytes</span><span class="w"> </span><span class="ss">(</span><span class="nv">pair</span><span class="w"> </span><span class="nv">address</span><span class="w"> </span><span class="nv">operation</span><span class="ss">))</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">SWAP</span><span class="c1">;</span>
<span class="w"> </span><span class="nv">PAIR</span>
<span class="w"> </span>}
</code></pre></div>
<p>And its initial storage is:</p>
<div class="highlight"><pre><span></span><code>Left { UNPACK (lambda bytes (pair address operation));
ASSERT_SOME;
LEFT address;
PUSH mutez 0;
NONE key_hash;
CREATE_CONTRACT {<<script_of_the_link_contract>>};
SWAP;
PAIR}
</code></pre></div>
<ul>
<li>In Carthage, the cost of each transfer in this chain of calls is 27817 gu. We can build a chain of 36 transfers and one origination before reaching the per-operation limit of 1,040,000 gu.</li>
<li>In Delphi, the cost for each transfer is about 4779 gu. We can build a chain of 217 transfers and one origination before reaching the limit of 1,040,000 gu.</li>
</ul>
<p>Thus, <strong>Delphi lets us grow the chain by a factor of almost 6</strong>.</p>
<h1 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h1>
<p>The Delphi protocol reduces gas costs and allows more complex smart contracts to be deployed and executed. The gas gains vary depending on the source of gas consumption. We measured this using scripts tailored to reach the gas limit in different ways. The results in short are:</p>
<table>
<thead>
<tr>
<th>Gas source</th>
<th>Carthage consumed gas</th>
<th>Delphi consumed gas</th>
<th>Ratio</th>
</tr>
</thead>
<tbody>
<tr>
<td>Writing to disk</td>
<td>535741 gu / 32 KiB</td>
<td>555330.160 gu / 2MiB</td>
<td>61.7</td>
</tr>
<tr>
<td>Reading from disk</td>
<td>344481 gu / 32 KiB</td>
<td>520045.437 gu / 2MiB</td>
<td>42.4</td>
</tr>
<tr>
<td>Parsing</td>
<td>248168 gu</td>
<td>6003.655 gu</td>
<td>41.4</td>
</tr>
<tr>
<td>Computation</td>
<td>996143 gu</td>
<td>230406.664 gu</td>
<td>4.32</td>
</tr>
<tr>
<td>Arithmetic</td>
<td>1039686 gu</td>
<td>389054.329 gu</td>
<td>2.67</td>
</tr>
<tr>
<td>Recursion</td>
<td>13178 gu</td>
<td>2871.559 gu</td>
<td>4.59</td>
</tr>
<tr>
<td>Inter-contract call</td>
<td>27817 gu</td>
<td>4779.106 gu</td>
<td>5.82</td>
</tr>
</tbody>
</table>
<p>So to sum up:</p>
<ol>
<li>Delphi enjoys a <a href="#parsing">significant reduction in parsing costs</a>: thanks to the refinement of the gas model for parsing, most contracts should see their parsing gas reduce by a factor of about 40, compared to Carthage, making it more practical to deploy and operate on larger contracts. </li>
<li>Similarly <a href="#io">the cost of large disk accesses (I/Os) is reduced</a> by a factor of about 60 for the largest disk writes.</li>
<li>Together, our improvements make <a href="#inter-contract-calls">inter-contract calls</a> cheaper by a factor of 6 in our tests.</li>
<li><a href="#contract-interpretation">Improvements in contract interpretation costs</a> exist though they are less striking because the cost model was quite precise already in Babylon (the version preceding Carthage). Yet, even here we can see quite significant improvements thanks to Delphi’s <a href="#improved-gas-accounting">optimisations in the gas costs of the Michelson interpreter’s gas accounting system</a>.</li>
</ol>
<p>In a future blog post, we will measure the gas gains observed on a collection of real-world contracts that are either already popular on mainnet or about to be launched.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:2">
<p><span class="dquo">“</span>Executes” here means “interprets”, in the sense of “an interpreter”. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>Returning a (non-empty) list of operations is sometimes also referred to as <em>emitting</em> or <em>applying</em> them. <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:4">
<p>We distinguish between a recursion when a lambda calls itself, and a recursive contract call when a contract calls itself (with all the costs this entails). Here, we’re dealing just with the first kind of recursion. <a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Delphi, the Latest Tezos upgrade, is live!2020-11-12T16:00:00+01:002020-11-12T16:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2020-11-12:/delphi-the-latest-tezos-upgrade-is-live.html<p>Summary:</p>
<p><em>This is a joint post from Nomadic Labs, Metastate and Marigold.</em></p>
<p>We’re very happy to announce that the vote on the “Delphi” upgrade to the Tezos network passed a few hours ago (around 13:00 <span class="caps">GMT</span> on 12 November 2020.) The upgrade went live immediately afterwards at block …</p><p>Summary:</p>
<p><em>This is a joint post from Nomadic Labs, Metastate and Marigold.</em></p>
<p>We’re very happy to announce that the vote on the “Delphi” upgrade to the Tezos network passed a few hours ago (around 13:00 <span class="caps">GMT</span> on 12 November 2020.) The upgrade went live immediately afterwards at block 1,212,417. </p>
<p>An informal blog post describing Delphi is <a href="https://research-development.nomadic-labs.com/delphi-official-release.html">here</a>, and a changelog of everything that went into Delphi is <a href="https://research-development.nomadic-labs.com/delphi-changelog.html#007-delphi-changelog">here</a>. Most prominently, Delphi makes substantial improvements to the performance of the Michelson interpreter and to the gas model, and also reduces storage costs by a factor of four to reflect improvements in the underlying storage layer. We are very pleased that Delphi has gone live; it will dramatically loosen the gas constraints currently experienced by the creators of sophisticated smart contracts operating on Tezos, enabling ever more interesting applications of the system.</p>
<p>As described in <a href="https://forum.tezosagora.org/t/regular-scheduling-for-our-tezos-proposals/2331">this</a> other blog post, we intend to continue producing a regular cadence of upgrade proposals to the Tezos network over time; the next, which will likely be named “Edo”, should appear very soon. </p>
<p>We are very excited about the features being proposed in Edo, and hope to announce them in the near future.</p>The case of mixed forks in Emmy+2020-10-26T12:00:00+01:002020-10-26T12:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2020-10-26:/the-case-of-mixed-forks-in-emmy.html<p><strong>Note</strong>: <em>This analysis was done with the help of Bruno Blanchet
(Inria). The interested reader can experiment with <a href="https://gitlab.com/nomadic-labs/emmyplus-experiments/-/tree/mixed-forks">our code used in
the
analysis</a>.
As in the <a href="/analysis-of-emmy.html">previous analysis</a>, we do not
present any security proofs.</em></p>
<p>This is the fourth in a series of posts on Emmy<sup>+</sup>:</p>
<ol>
<li>After our <a href="/analysis-of-emmy.html">initial analysis</a>,</li>
<li>recently <a href="/on-defending-against-malicious-reorgs-in-tezos-proof-of-stake.html">revisited</a> and</li>
<li><a href="/emmy-in-the-partial-synchrony-model.html">extended to the partial synchronous network model</a>,</li>
<li>we now consider so-called “mixed forks”.</li>
</ol>
<p>So far, we assumed that malicious bakers wanted to undo a transaction. In
this post, we consider instead that they want to maintain
a (malicious) fork for as long as possible.</p>
<p>We provide experimental data that
such scenarios do not have a significant impact on the <a href="/emmy-in-the-partial-synchrony-model.html">previous
analysis</a>, which thus remains robust in the presence of this kind of attack.</p>
<h2 id="mixed-forks"><a class="toclink" href="#mixed-forks">Mixed forks</a></h2>
<p><span class="math">\(\newcommand\f[1]{\mathit{#1}}\)</span>
<span class="math">\(\newcommand\edelay{\f{delay}}\)</span>
<span class="math">\(\newcommand\edelaydelta{\edelay_{\Delta}}\)</span>
<span class="math">\(\newcommand\ie{\f{ie}}\)</span>
<span class="math">\(\newcommand\dpp{\f{dp}}\)</span>
<span class="math">\(\newcommand\de{\f{de}}\)</span>
<span class="math">\(\newcommand\dpde{\f{dpde}}\)</span>
<span class="math">\(\newcommand\od{\tau}\)</span>
<span class="math">\(\newcommand\emmy{\f{Emmy^+}}\)</span>
<span class="math">\(\newcommand\hb{H}\)</span>
<span class="math">\(\newcommand\cb{C}\)</span>
<span class="math">\(\newcommand\pr{\mathtt{Pr}}\)</span>
<span class="math">\(\newcommand\pprio[1]{\mathbb{P}_{prio}(#1)}\)</span>
<span class="math">\(\newcommand\pendo[1]{\mathbb{P}_{endo}(#1)}\)</span>
<span class="math">\(\newcommand\pdiff[1]{\mathbb{P}_{\Delta}(#1)}\)</span>
<span class="math">\(\newcommand\difff[1]{\f{diff}_{=1}(#1)}\)</span>
<span class="math">\(\newcommand\diff[1]{\f{diff}_{>1}(#1)}\)</span>
<span class="math">\(\newcommand\diffl[1]{\f{diff}_{\ell}(#1)}\)</span>
<span class="math">\(\newcommand\difffp[1]{\f{diff^{\leftarrow}}_{=1}(#1)}\)</span>
<span class="math">\(\newcommand\diffp[1]{\f{diff^{\leftarrow}}_{>1}(#1)}\)</span>
<span class="math">\(\newcommand\secu{\eta}\)</span>
<span class="math">\(\newcommand\barpc{\bar{\pc}}\)</span>
<span class="math">\(\newcommand\barph{\bar{p}^\star}\)</span>
<span class="math">\(\newcommand\tsh{t^\star}\)</span>
<span class="math">\(\newcommand\tsc{t}\)</span>
<span class="math">\(\newcommand\diffg{\f{diff}}\)</span>
<span class="math">\(\newcommand\pc{p}\)</span>
<span class="math">\(\newcommand\maxp{\f{max\_p}}\)</span>
<span class="math">\(\newcommand\ph{p^\star}\)</span>
<span class="math">\(\newcommand\diffgns{\f{diff}^{\;ns}}\)</span>
<span class="math">\(\newcommand\diffgs{\f{diff}^{\;s}}\)</span>
<span class="math">\(\newcommand\cchain{\f{ch}_\cb}\)</span>
<span class="math">\(\newcommand\hchain{\f{ch}_\hb}\)</span></p>
<p>For readability we introduce some notation: we use <span class="math">\(\hb\)</span>, resp. <span class="math">\(\cb\)</span>
to distinguish between honest and corrupt.</p>
<p>We can think
of several dishonest bakers <span class="math">\(\cb_i\)</span> as being a single ‘composite’ dishonest baker <span class="math">\(\cb\)</span> having
as stake fraction the sum of the stake fractions of <span class="math">\(\cb_i\)</span>, and similarly for the honest bakers.</p>
<p>So for simplicity, we reason henceforth using a pair of an honest
baker <span class="math">\(\hb\)</span> and a dishonest baker <span class="math">\(\cb\)</span> acting as an adversary.</p>
<p>In our previous analyses (as above) we assumed
<span class="math">\(\cb\)</span> wants to undo a transaction, so:</p>
<ol>
<li><span class="math">\(\cb\)</span> bakes a
secret chain <span class="math">\(\cchain\)</span> and</li>
<li>once <span class="math">\(\cchain\)</span> is faster than <span class="math">\(\hb\)</span><span class="quo">‘</span>s chain <span class="math">\(\hchain\)</span>, <span class="math">\(\cb\)</span> reveals it, and</li>
<li>honest <span class="math">\(\hb\)</span> adopts <span class="math">\(\cchain\)</span> and bakes on it thenceforth.</li>
</ol>
<p>Now, suppose <span class="math">\(\cb\)</span> wants to maintain the system in a forked state<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup> as long as possible. So:</p>
<ol>
<li><span class="math">\(\cb\)</span> still bakes a secret chain <span class="math">\(\cchain\)</span>,</li>
<li><span class="math">\(\cb\)</span> still reveals it once it is faster than <span class="math">\(\hchain\)</span>, and</li>
<li><span class="math">\(\hb\)</span> still adopts <span class="math">\(\cchain\)</span> and bakes on it — but now,</li>
<li><span class="math">\(\cb\)</span> continues to secretly bake, this time on <span class="math">\(\hchain\)</span>.</li>
</ol>
<p>The roles of the two chains are thus swapped, and
that’s why we talk about <strong>mixed forks</strong>: the same chain may be <span class="math">\(\hb\)</span><span class="quo">‘</span>s chain at
some point and <span class="math">\(\cb\)</span><span class="quo">‘</span>s chain at some other point, and this swapping of roles may happen more than once.</p>
<p>We illustrate such a scenario in Figure 1 below, where:</p>
<ul>
<li>Blocks (and their timestamps) baked by <span class="math">\(\hb\)</span> are identified by the “<span class="math">\(^\star\)</span>” superscript.</li>
<li>Blocks baked by <span class="math">\(\cb\)</span> are in red.</li>
<li>The hashed red pattern denotes a block that <span class="math">\(\cb\)</span> bakes but does not reveal.</li>
<li>A block index represents the level at which the block is baked.</li>
<li>The number in each block denotes the block’s priority.</li>
</ul>
<p>We assume <span class="math">\(\cb\)</span> has enough stake and chance to have consecutive 0 priorities for the blocks following
<span class="math">\(b_1\)</span>, so that <span class="math">\(\cb\)</span> can bake those blocks sooner than <span class="math">\(\hb\)</span>.</p>
<p>Figure 1 shows that:</p>
<ol>
<li>
<p>At level 1, <span class="math">\(\cb\)</span> starts a fork from <span class="math">\(b^\star_0\)</span> by baking <span class="math">\(b_1\)</span>;
<span class="math">\(\cb\)</span> cannot reveal <span class="math">\(b_1\)</span> because <span class="math">\(b_1\)</span> has a larger timestamp than
<span class="math">\(b^\star_1\)</span> baked by <span class="math">\(\hb\)</span>.</p>
</li>
<li>
<p>At level 2, <span class="math">\(\cb\)</span> double-bakes (bakes on its own secret chain <em>and</em>
on the chain of <span class="math">\(\hb\)</span>) and reveals <span class="math">\(b_2\)</span>, the block baked on its own
secret chain before <span class="math">\(\hb\)</span> bakes at this level (the block <span class="math">\(b^\star_2\)</span>
which <span class="math">\(\hb\)</span> would have baked had it not been <span class="math">\(b_2\)</span> is in gray, drawn
with dashed lines).</p>
</li>
<li>
<p>At level 3, <span class="math">\(\cb\)</span> bakes <span class="math">\(b_3\)</span> on top of <span class="math">\(b'_2\)</span> while <span class="math">\(\hb\)</span> bakes <span class="math">\(b^\star_3\)</span>
on top of <span class="math">\(b_2\)</span>; <span class="math">\(\hb\)</span> does so because <span class="math">\(\hb\)</span> swapped chains once <span class="math">\(\cb\)</span>
revealed <span class="math">\(b_2\)</span>: at that point, <span class="math">\(\cb\)</span><span class="quo">‘</span>s alternative chain <span class="math">\(b_0^\star b_1b_2\)</span> is
longer than <span class="math">\(\hb\)</span><span class="quo">‘</span>s chain <span class="math">\(b_0^\star b_1^\star\)</span>.</p>
</li>
<li>
<p>At level 4, <span class="math">\(\cb\)</span> bakes <span class="math">\(b_4\)</span> on top of <span class="math">\(b_3\)</span> and reveals it before
<span class="math">\(\hb\)</span> bakes at this level (note that we are in a similar situation
as that at level 2).</p>
</li>
<li>
<p>At level 5, <span class="math">\(\hb\)</span> bakes <span class="math">\(b^\star_5\)</span> on top of <span class="math">\(b_4\)</span>: once <span class="math">\(\cb\)</span>
revealed <span class="math">\(b_4\)</span>, <span class="math">\(\hb\)</span> swapped again chains because the alternative
chain is longer than the one on which <span class="math">\(\hb\)</span> was baking. This is the
same reasoning as at level 3. Note that <span class="math">\(\hb\)</span> bakes <span class="math">\(b^\star_5\)</span> on
the chain <span class="math">\(\hb\)</span> initially started baking on.</p>
</li>
</ol>
<p><img alt="Image" src="images/emmyplus_mixed_forks/mixed_forks.png"></p>
<p><span class="math">\(\cb\)</span> could in theory continue baking on both chains and force <span class="math">\(\hb\)</span> to swap
chains repeatedly, assuming <span class="math">\(\cb\)</span> receives priorities and endorsement slots
to allow it. We would thus like to quantify to what extent can <span class="math">\(\cb\)</span>
do that: how often and for how long.</p>
<h3 id="probabilities-of-mixed-forks"><a class="toclink" href="#probabilities-of-mixed-forks">Probabilities of mixed forks</a></h3>
<p>To experimentally show that the probability that <span class="math">\(\cb\)</span> can maintain a fork decreases (sufficiently rapidly) as fork length increases, we revisit the scenario <a href="/emmy-in-the-partial-synchrony-model.html#forks-starting-now">“forks starting
now”</a>. We
consider the following question:</p>
<blockquote>
<p>How long can <span class="math">\(\cb\)</span> maintain the system in a forked state?</p>
</blockquote>
<p>To answer this, we use an approach similar to the one for computing
the probabilities of forks in the non-mixed forks case. We presented
the methodology behind the analysis in a <a href="/on-defending-against-malicious-reorgs-in-tezos-proof-of-stake.html#on-the-methodology">previous
post</a>. However,
the key aspects needing careful consideration are:</p>
<ul>
<li>how to identify when a swap is feasible and advantageous for <span class="math">\(\cb\)</span>, and</li>
<li>how the update the time difference between chains at
the moment of a swap.</li>
</ul>
<p>Using our updated analysis, we plot the probabilities of mixed forks
and compare them with the probabilities of non-mixed forks as already
computed in the <a href="/emmy-in-the-partial-synchrony-model.html#forks-starting-now">previous
analysis</a>). In
this experiment we assumed that the network is synchronous.<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup></p>
<p><img alt="Image" src="images/emmyplus_mixed_forks/mixed_forks_starting_now.png"></p>
<p>The plot suggests that, for decent attacker stake fractions such as
.2, or even .3, the expected number of confirmations in the mixed case
is only a bit larger than in the non-mixed case. For instance, for .3
attacker stake fraction, one needs to wait a priori for 20 confirmations instead of
16. The expected number of confirmations gets larger for higher
attacker stakes: for .4, the expected number of confirmations is 85, vs.
67 for the non-mixed case.</p>
<p>To conclude, our experiments show that, even if <span class="math">\(\cb\)</span> tries to
repeatedly make <span class="math">\(\hb\)</span> swap chains, <span class="math">\(\cb\)</span> cannot do this for too long:
the number of expected confirmations when considering “mixed forks” is
only slightly higher than without them.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>This post considers specifically the case that <span class="math">\(\cb\)</span> tries to maintain <em>one</em> fork. Of course we could generalise this to multiple forks, but our simulations on a simplified model suggest that one fork is already a good approximation to the more general case. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>We recall that <span class="math">\(\Delta = 30\)</span> represents the case when the network is <a href="/emmy-in-the-partial-synchrony-model.html#observation-1">synchronous</a>. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Regular Scheduling For Our Tezos Proposals2020-10-21T00:00:00+02:002020-10-21T00:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2020-10-21:/regular-scheduling-for-our-tezos-proposals.html<p>The teams at <a href="https://www.nomadic-labs.com/">Nomadic Labs</a>, <a href="https://metastate.dev/">Metastate</a>, Marigold, and <a href="https://dailambda.jp/">DaiLambda</a> have participated in a number of joint protocol proposals for Tezos; some of us have been working on the code since the original launch of the Tezos network, and have been involved with updates from Athens through the recent Delphi proposal. Over time, we have gained more and more experience and have learned what practices seem to work best for updates to the Tezos ecosystem.</p>
<p>Up until recently, we have generally focused on releasing proposals when a particular pre-determined set of features have become stable. As the community has matured, and as we have gained experience, we have developed the belief that it is better to release proposals on a regularly scheduled basis instead.</p>
<p>The rest of this blog post describes our reasoning.</p>
<p>In software development, teams are often tempted to release a new version of their code only after a particular set of features is complete. The problem with this approach is that it often leads to releases occurring at longer and longer intervals.</p>
<p>The syndrome tends to work this way: a particular feature is on the must-have list for a release, but is delayed. A developer of another feature worries that because the gap between releases has increased, it is important that their feature be incorporated before the next release or it may not see the light of day for some time. The work to incorporate this change then delays the release further.</p>
<p>Then, other developers notice that releases are happening at longer and longer intervals, and they in turn become more and more worried that if they don’t get their features into the next release, it might be a very long time before users see it.</p>
<p>Soon, a vicious cycle is in progress where releases happen further and further apart, first being delayed by months, and then by years.</p>
<p>One remedy that many projects have found for this problem is straightforward: release code regularly, at scheduled intervals rather than when particular features are complete.</p>
<p>The reason this works is also straightforward. If trains leave a station every ten minutes, few people will be worried about missing a train; there will always be another a few minutes later. However, if trains leave without a fixed schedule and at long intervals, people become very paranoid that when the current train leaves, there might not be another for quite some time.</p>
<p>An important feature of Tezos is its ability to evolve without breaking its own rules. Tezos’ on-chain governance provides a neutral mechanism for stakeholders to coordinate and agree on updates to the protocol and its implementation.</p>
<p>Like many blockchain projects, Tezos is an open-source project with contributors from all over the world but, unlike other projects, it provides a mechanism through which anyone can credibly advance proposals without facing the typically insurmountable hurdle of coordinating a hard-fork.</p>
<p>Our teams, alongside many other contributors around the world, have historically collaborated on making ambitious proposals for the Tezos project. Going forward, we have decided to collaborate to submit protocol proposals every few months, which is the interval permitted by the Tezos on-chain amendment process. These will incorporate code that has been completed to our satisfaction by the scheduled date, rather than our introducing new proposals only when a predetermined set of features are finished.</p>
<p>The capacity to evolve is a key distinguishing feature of Tezos. With a scheduled releases approach, protocol proposals will occur with a much more steady cadence. We believe that this will, in turn, allow Tezos to gain the features developers and users have requested on a more dependable timeline.</p>Meanwhile at Nomadic Labs #92020-10-20T15:00:00+02:002020-10-20T15:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2020-10-20:/meanwhile-at-nomadic-labs-9.html<p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>It’s been a while since we published a post in our <a href="https://research-development.nomadic-labs.com/blog.html?category=meanwhile">meanwhile series</a>, and as always we’ve been working hard behind the scenes to improve the Tezos ecosystem. </p>
<p>August marked a milestone: we launched <a href="https://tezos.gitlab.io/introduction/test_networks.html#dalphanet"><strong>Dalphanet</strong></a>, a dedicated test network designed to examine features from all developers involved in submitting the more extensive and long-awaited protocol proposal, including Sapling, a new protocol environment, among other improvements. For more information on Dalphanet see:</p>
<ul>
<li><a href="https://forum.tezosagora.org/t/dalphanet-was-launched/2089">Agora Forum</a></li>
<li><a href="https://www.reddit.com/r/tezos/comments/hzydjv/community_update/">Reddit</a></li>
</ul>
<p>On August 19th, Raphaël Cauderlier attended the <a href="https://tquorum.com/the-series/2020-08-19/tezos-townhall">Tezos Town Hall</a>.
The main topics were governance and technical processes within the Tezos ecosystem.
<a href="https://www.tezosagora.org/learn">Tezos’ governance</a> is uniquely innovative and flexible.
The discussion shifted to high gas fees which led to the proposal of <a href="https://forum.tezosagora.org/t/delphi/2150">Delphi</a>, an intermediate proposal to lower gas costs when interacting with smart contracts on the Tezos blockchain.</p>
<p>In conclusion, <a href="https://gitlab.com/galfour">Gabriel Alfour</a>, <a href="https://metastate.dev/">Metastate</a> and <a href="https://www.nomadic-labs.com/">Nomadic Labs</a> announced <a href="https://forum.tezosagora.org/t/delphi/2150">Delphi</a>, a smaller upgrade to the Tezos blockchain focused on lowering gas fees. Also, Delphi reduces storage costs by a factor of four, to reflect improvements in the underlying storage layer.</p>
<p>The public baker <a href="https://stakery.io/">Stakery</a> was first to inject the Delphi protocol to the Tezos blockchain. We are currently in the testing period, and the promotion period will start very soon. Follow this <a href="https://www.tezosagora.org/">link to Agora</a> to track the progress in real-time. Since the injection, we have also been working on finalizing a proposal for 008.</p>
<p>We also released <a href="https://tezos.gitlab.io/releases/version-7.html#">Version 7.4</a>, enabling the community to participate in <a href="https://forum.tezosagora.org/t/delphinet-has-been-launched-version-7-4-was-released/2165">Delphinet</a>, a test network that we spawned for the recently injected Delphi proposal.<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup></p>
<p>Nomadic Labs is pleased to support another promising project:
<a href="https://www.sword-group.com/en/news/tezos-digisign-an-open-source-solution/">Tezos DigiSign</a>, an open source solution to digitally sign, store and verify documents on the Tezos blockchain, launched by <a href="https://www.sword-group.com/en/">Sword Group</a>.
Read more in <a href="https://www.nomadic-labs.com/download/PR-Tezos-Digisign.pdf">Tezos Digisign press release</a>.</p>
<p>In mid-September we: </p>
<ul>
<li>published our latest press release: <a href="https://www.nomadic-labs.com/download/cp.pdf">Tezos selected by Societe Generale/Forge for its central banking digital currency experiment</a>;</li>
<li>published a new blog post on <a href="https://research-development.nomadic-labs.com/on-defending-against-malicious-reorgs-in-tezos-proof-of-stake.html">defending against malicious reorgs in Tezos proof of stake</a>;</li>
<li>attended September’s <a href="https://hopin.to/events/session-11-tezos-town-hall">Tezos Town Hall</a> with topics ranging from Tezos core development to the technical decision-making in the Tezos ecosystem; and </li>
<li>co-organized a <a href="https://twitter.com/LabosNomades/status/1312002138646573057?s=20">scientific workshop</a> with <a href="https://www.inria.fr/en">Inria</a>. As part of our collaboration we held presentations and discussions on research <span class="amp">&</span> tools within the Tezos ecosystem.</li>
</ul>
<p>In October we:</p>
<ul>
<li>published two blog posts: <ul>
<li><a href="https://research-development.nomadic-labs.com/emmy-in-the-partial-synchrony-model.html">Emmy<sup>+</sup> in the partial synchrony model</a> describes Emmy<sup>+</sup>, going beyond the synchronous network model, and </li>
<li><a href="https://research-development.nomadic-labs.com/dexter-decentralized-exchange-for-tezos-formal-verification-work-by-nomadic-labs.html">Dexter: Decentralized exchange for Tezos, formal verification work by Nomadic Labs</a> goes into detail on how we verified a functional specification of Dexter’s core smart contract using Mi-Cho-Coq;</li>
</ul>
</li>
<li>uploaded our <a href="https://nomadic-labs.com/internships.html">internship catalog</a> featuring ten open positions; and</li>
<li>announced <a href="https://forum.tezosagora.org/t/new-release-candidate-version-8-0/2307">a new release candidate</a>: <a href="http://tezos.gitlab.io/releases/version-8.html">Version 8.0</a>.</li>
</ul>
<p>We look forward to the future and continued growth of users, developers <span class="amp">&</span> builders within the Tezos ecosystem.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p><a href="https://forum.tezosagora.org/t/delphinet-has-been-launched-version-7-4-was-released/2165">Delphinet</a> is not to be confused with <a href="https://tezos.gitlab.io/introduction/test_networks.html#dalphanet">Dalphanet</a>! <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Dexter: Decentralized exchange for Tezos, formal verification work by Nomadic Labs2020-10-14T10:30:00+02:002020-10-14T10:30:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2020-10-14:/dexter-decentralized-exchange-for-tezos-formal-verification-work-by-nomadic-labs.html<p>The Dexter smart contract is the cornerstone of the Dexter. We outline a formal verification of its functional specification, in Coq.</p><p align="center" style="margin: 20px 0">
<img style="margin-right: 60px" src="images/dexter-verification/dexter.jpg" />
</p>
<p>On September 30th, <a href="https://camlcase.io/">camlCase</a> released
<a href="https://dexter.exchange/">Dexter</a>. </p>
<p>Dexter is a smart contract empowering decentralized exchange, performing the same function for tez as
<a href="https://uniswap.org/">Uniswap on Ethereum</a> does for ether; it enables
trade between tez and any other token implemented by <a href="https://assets.tqtezos.com/docs/token-contracts/fa12/1-fa12-intro/">an <span class="caps">FA1</span>.2
contract</a>.</p>
<p>To this day, the following efforts have been undertaken to ensure Dexter’s quality, including:</p>
<ul>
<li>a security <a href="https://github.com/trailofbits/publications/blob/master/reviews/dexter.pdf">audit</a> by <a href="https://www.trailofbits.com/">Trail of Bits</a>, </li>
<li><a href="https://gitlab.com/camlcase-dev/dexter/-/tree/master/morley/tests">extensive property-based testing</a> by camlCase, and</li>
<li>a formal verification of the functional specification of Dexter’s underlying smart contract by Nomadic Labs, using the <a href="https://coq.inria.fr/">Coq proof assistant</a> with the <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/">Mi-Cho-Coq framework</a> — the topic of this blog post.</li>
</ul>
<p>Checking smart contracts is never about attaining certainty; it’s about clarifying what we do and do not know.
Therefore, our efforts increase our confidence in the Dexter smart contract, but do not make errors impossible. In this blog post we give:</p>
<ul>
<li>an outline of what we have <strong>proved</strong> about Dexter, and also </li>
<li>what we did <strong>not prove</strong>. </li>
</ul>
<h1 id="what-is-dexter-and-fa12"><a class="toclink" href="#what-is-dexter-and-fa12">What is Dexter and <span class="caps">FA1</span>.2?</a></h1>
<p>Dexter is a <a href="https://gitlab.com/camlcase-dev/dexter/">smart contract</a>, with an accompanying <a href="https://app.dexter.exchange/">decentralized app</a> enabling decentralized token exchange on the Tezos blockchain.
Users call Dexter’s smart contract to trade between tez and other tokens, without the intervention of a central authority.
For instance:</p>
<ul>
<li>tez can be exchanged to <a href="https://tzbtc.io/">tzBTC</a>, a token that wraps bitcoin, and</li>
<li>tez can be exchanged to <a href="https://usdtz.com/">USDtez</a>, a <span class="caps">USD</span>-pegged stable-coin token<sup id="fnref:4"><a class="footnote-ref" href="#fn:4">4</a></sup>.</li>
</ul>
<p>More precisely, Dexter is compatible with any <span class="caps">FA1</span>.2 token<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>. <a href="https://assets.tqtezos.com/docs/token-contracts/fa12/1-fa12-intro/"><span class="caps">FA1</span>.2</a> is a
<a href="https://gitlab.com/tzip/tzip/-/blob/master/proposals/tzip-7/tzip-7.md">standard</a>
for smart contracts implementing tokens on the Tezos blockchain. It
specifies a minimal set of entrypoints for the contract to provide and
how these entrypoints must behave. It is similar to the <a href="https://eips.ethereum.org/EIPS/eip-20"><span class="caps">ERC20</span></a>
standard for Ethereum.</p>
<h1 id="formally-verifying-a-functional-specification-of-dexter"><a class="toclink" href="#formally-verifying-a-functional-specification-of-dexter">Formally verifying a functional specification of Dexter</a></h1>
<p>Nomadic Labs has experience of applying formal verification to smart contracts, most notably through our
formal verification of the <a href="https://arxiv.org/abs/1909.08671">multisig</a> and
<a href="https://research-development.nomadic-labs.com/formally-verifying-a-critical-smart-contract.html">the spending limit</a>
contracts. We brought this experience to bear on Dexter as follows: </p>
<ul>
<li>We built an <strong><a href="https://gitlab.com/camlcase-dev/dexter/-/blob/master/docs/dexter-informal-specification.md">informal, functional, specification</a></strong> (in collaboration with camlCase);</li>
<li><strong><a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/blob/dexter-verification/src/contracts_coq/dexter_spec.v">formalized</a></strong> this specification in Coq; and</li>
<li><strong><a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/blob/dexter-verification/src/contracts_coq/dexter_verification_main.v">certified</a></strong> that the Dexter smart contract satisfies the formal
specification, using a machine-checked proof in Coq.</li>
</ul>
<p>The informal specification clearly and comprehensively describes the Dexter smart contract’s desired functional behaviours in natural language. This does three important things:</p>
<ol>
<li>It’s a clear, human-readable, functional specification of the contract. A functional specification defines the contract’s desired, immediate output, in terms of updated storage and emitted operations, for each input. </li>
<li>The act of rigorously working through the specification improved the quality of the final code in a virtuous engineering cycle which ironed out details and covered corner cases.</li>
<li>We could then convert this human-readable specification into a formalization suitable for machine checking in Coq. </li>
</ol>
<p>This work, while considerable and consequential, cannot be a panacea or
a silver bullet. The following aspects has only been checked in the <a href="https://github.com/trailofbits/publications/blob/master/reviews/dexter.pdf">audit</a>, and through <a href="https://gitlab.com/camlcase-dev/dexter/-/tree/master/morley/tests">testing</a>, but have not been subject to a formal study:</p>
<ul>
<li>we cannot guarantee the <a href="#the-soundness-of-dexters-specification-should-be-verified">soundness of the specification</a>;</li>
<li>we have not formally verified Dexter’s <a href="#dexters-inter-contract-communication-should-be-verified">inter-contract communication</a>; and finally</li>
<li>we have not formally verified the <a href="#temporal-properties-of-dexter-should-be-verified">temporal properties</a> of Dexter.</li>
</ul>
<p>This is subject to future work. We discuss each of these limitations
in more detail <a href="#future-work">below</a>.</p>
<p>But, within these limitations: we know what the contract
does, we checked the design space in detail — and furthermore, we
<em>proved</em> it.</p>
<p>Specified, certified code in this style is also an important long-term investment in maintainability and security.
For example it is future work to verify high-level, economic properties such as <a href="https://arxiv.org/pdf/1911.03380.pdf">non-depletion</a>, and to do this we will not have to look at Dexter’s source code:
we can read the human-readable specification and/or reason from the formal specification, and our certification ensures the conclusions will hold of the implementation.</p>
<h2 id="methodology"><a class="toclink" href="#methodology">Methodology</a></h2>
<p>Our development followed a policy where the contract
is specified, implemented, and verified by three distinct teams:</p>
<ul>
<li>the <strong>specifiers</strong>,</li>
<li>the <strong>implementors</strong>, and</li>
<li>the <strong>verifiers</strong>.</li>
</ul>
<p>This policy forces the
teams to verify each other’s work, and thus increases overall
confidence in the development.</p>
<p>For example, if the verification team spots a divergence between the code
and the specification, they can’t just change one to suit the other:
they have to communicate with the
implementors and specifiers to agree on an intended behavior,
and update the code and/or the specification accordingly. </p>
<p>For Dexter, we have had:</p>
<ul>
<li>one specifier who worked with camlCase with the informal
specification, and then worked with the formalization of that specification;</li>
<li>camlCase that implemented the code of Dexter; and</li>
<li>two verifiers that proved the contract against the formal specification.</li>
</ul>
<p>We followed this policy for most of the
entrypoints, but time constraints meant the specifier also
participated in the verification of 3 out of 11 entrypoints.</p>
<h2 id="time-investment"><a class="toclink" href="#time-investment">Time investment</a></h2>
<p>Work with the informal and formal specification of Dexter has been
ongoing part-time since November 2019. In total, the collaborating on
the informal specification and developing the formal specification
took 2 person-weeks from Nomadic Lab’s side. The final proof took
three verification engineers four person-weeks to write.</p>
<h1 id="verification-framework"><a class="toclink" href="#verification-framework">Verification framework</a></h1>
<p>The functional specification of Dexter has been verified in the
<a href="https://coq.inria.fr/">Coq proof assistant</a> using the
<a href="https://gitlab.com/nomadic-labs/mi-cho-coq/">Mi-Cho-Coq framework</a>.</p>
<p>Coq is a rich environment, in which we can directly state and prove a wide variety of mathematical properties.
In particular we can specify programs, program executions, and properties of program executions — including correctness with respect to a specification — all in a single theorem-proving environment.</p>
<p>Likewise Mi-Cho-Coq embeds Michelson smart contracts in Coq, so we can specify Michelson contracts and verify their properties, all in Coq — for instance, that a contract can’t fail or run out of tokens.</p>
<p>We do not reason directly on the Mi-Cho-Coq interpreter, since implementation details of how scripts are interpreted (think: stack manipulation), are irrelevant. Instead, we use Mi-Cho-Coq’s <a href="https://en.wikipedia.org/wiki/Predicate_transformer_semantics">weakest-precondition calculator</a>, which
translates a contract and its execution into a logical proposition. To prove properties of the contract, we just need to prove properties of the corresponding logical proposition.</p>
<h1 id="findings"><a class="toclink" href="#findings">Findings</a></h1>
<p>Developing a chain of formal spec / informal spec / implemented code, and checking that the three match up, catches errors. Below, we detail the errors Nomadic Labs found in the Dexter contract. These errors have been fixed in the version of Dexter running on mainnet.</p>
<h2 id="parameter-verification"><a class="toclink" href="#parameter-verification">Parameter verification</a></h2>
<p>Each entrypoint of Dexter specifies a set of conditions on the parameters that must hold to successfully execute the entrypoint.</p>
<ul>
<li>Sometimes the formal specification omitted parameter verifications that the informal specification said should be present. </li>
<li>Sometimes the code omitted parameter verifications that the informal specification said should be present (this can be a particular security issue).</li>
<li>Sometimes the informal specification omitted parameter verifications that the code implemented.</li>
</ul>
<h2 id="operation-order"><a class="toclink" href="#operation-order">Operation order</a></h2>
<p>Michelson contracts emit operations at the end of execution
(e.g. token transfers). In Dexter’s case, the operations are transfers
either: to the caller of the contract; to a <span class="caps">FA1</span>.2-compliant token contract;
or to another Dexter contract. When verifying Dexter we
found incoherencies in the order of operations emitted
by Dexter with respect to the specification.</p>
<p>For example: suppose a user wants to sell <span class="math">\(X\)</span> many tokens, for <span class="math">\(Y\)</span> many tez using the <code>tokenToXtz</code> entrypoint.
<a href="https://gitlab.com/camlcase-dev/dexter/-/blob/master/docs/dexter-informal-specification.md#operations-5">Dexter’s specification</a>
asserts that Dexter should </p>
<ol>
<li>transfer <span class="math">\(Y\)</span> tez to the caller, and only then</li>
<li>transfer <span class="math">\(X\)</span> tokens from the caller to Dexter through a call to the
token contract.</li>
</ol>
<p>It turned out that the implementation would emit the operations in
the reverse order: first 2, then 1. This was not an exploitable security issue — but it could have been, and such a violation of a
specification complicates integration of Dexter with other contracts. The following example demonstrates why.</p>
<p>Consider a scenario where this error was not fixed, and where the caller
is itself a smart contract. The developer of the calling contract
would assume from the specification that when their contract receives
the <span class="math">\(Y\)</span> tez from Dexter, it had not yet payed the outstanding <span class="math">\(X\)</span>
tokens. Whereas in fact, the incoherent implementation of Dexter
would have transferred the <span class="math">\(X\)</span> tokens from the caller before sending the
tez to the caller.</p>
<p>So the calling contract might think it had
more tokens available than the actual balance,
and in mistakenly attempting to spend them, there might be insufficient funds for the <span class="caps">FA1</span>.2
token transfer initiated by the token-to-tez trade.</p>
<p>This scenario demonstrates the importance of the finer details of a
specification, and ensuring that the implementation satisfies it. </p>
<h2 id="deadline-checks"><a class="toclink" href="#deadline-checks">Deadline checks</a></h2>
<p>Dexter’s entrypoints for adding and removing liquidity and for performing trades contain a deadline
verification; so if an order fails its deadline verification it should be rejected. </p>
<p>We found a small divergence between the specification of deadlines and
their implementation. The specification states
that the deadline must be greater than the current time
stamp — so an order handled <em>exactly</em> on
the deadline should be rejected.</p>
<p>However, in some entrypoints the check was inclusive: an order handled exactly on the deadline was accepted.
This does not seem to have been an exploitable security issue, but
the process of finding and fixing the issue illustrates how formal
verification traps errors that might otherwise sneak into production code.</p>
<h1 id="future-work"><a class="toclink" href="#future-work">Future work</a></h1>
<p>As with any formal verification, there are limits to our
development. Some are inherent to software verification, while others
stem from limitations of Mi-Cho-Coq:</p>
<h2 id="formal-verification-of-the-soundness-of-dexters-specification"><a class="toclink" href="#formal-verification-of-the-soundness-of-dexters-specification">Formal verification of the soundness of Dexter’s specification</a></h2>
<!-- what we'd like to do -->
<!-- context !-->
<p>By <strong>soundness</strong> we mean conformity to some “common sense”
economic properties; e.g.
that an attacker can’t remove the funds
of a liquidity provider without permission.</p>
<!-- [aj: why we cant ] -->
<p>This is simple garbage in, garbage out: we can check whether an
implementation satisfies its specification, but what if the specification’s wrong?
For instance, a specification’s author might inadvertently permit an
attack by which an attacker would remove funds that are not theirs, by simple human error or by not fully understanding the domain logic of the system. </p>
<!-- If Dexter's specification is unsound, and the code
faithfully implements that specification, then a verification engineer
proving the specification will only have unwittingly confirmed the
unsoundness of the smart contract.
-->
<h2 id="formal-verification-of-dexters-inter-contract-communication"><a class="toclink" href="#formal-verification-of-dexters-inter-contract-communication">Formal verification of Dexter’s inter-contract communication</a></h2>
<!-- what we would like to verify -->
<!-- context -->
<p>Michelson contracts communicate by emitting transfer operations at the
end of their execution. If the destination of the transfer is another
Michelson contract, then that too will be executed, and in turn it may emit
operations. Thus, a single contract’s execution may trigger a
chain of multiple contract calls.</p>
<!-- example -->
<p>For example consider Dexter’s entrypoint <code>tokenToXtz</code>, which exchanges
tokens for tez. It emits a transfer to the caller with the obtained tez, and a
transfer to the token contract that moves the sold tokens to
Dexter. We’d like to assert that after this operation
sequence, Dexter indeed gets the sold tokens.</p>
<!-- why we cant -->
<p>Currently we can’t do this in Mi-Cho-Coq: we can confirm that Dexter emits the corresponding
call to <span class="caps">FA1</span>.2, but Mi-Cho-Coq does not implement the semantics of inter-contract communication<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup>, so we cannot specify (and so cannot
verify) the final effect of the complete chain of contract calls.</p>
<h2 id="formal-verification-of-dexters-temporal-properties"><a class="toclink" href="#formal-verification-of-dexters-temporal-properties">Formal verification of Dexter’s temporal properties</a></h2>
<!-- what we would like to do -->
<!-- example -->
<p>Temporal properties concern the behavior of a system over time. For
instance: we’d like to verify that a user who adds liquidity to Dexter,
can always remove it later.</p>
<!-- why we cant -->
<p>This requires reasoning over all possible sequences
of contract calls in the blockchain, which is currently impossible in
Mi-Cho-Coq: first, because it requires adding the semantics of inter-contract
communication, as discussed above; and second, because it also requires modeling
properties of the block chain, such that the level and timestamp of each block
in the chain is strictly increasing.</p>
<h1 id="results"><a class="toclink" href="#results">Results</a></h1>
<p>The proof of the contract is available in <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/merge_requests/71">merge request
!71</a>
of the <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/">Mi-Cho-Coq repository</a>.</p>
<p>Excluding comments and blank lines<sup id="fnref:3"><a class="footnote-ref" href="#fn:3">3</a></sup> the development is 2983 = 272 + 775 + 919 + 1017 lines of Coq code, as follows:</p>
<ul>
<li>importing the contract into Mi-Cho-Coq takes 272 lines;</li>
<li>the specification is 775 lines long;</li>
<li>the proofs of the main theorems (see <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/blob/dexter-verification/src/contracts_coq/dexter_verification.v">here</a> for the proof of individual entrypoints and <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/-/blob/dexter-verification/src/contracts_coq/dexter_verification_main.v">here</a> for the proof of the full contract) span 919 lines total;</li>
<li>additionally, the proofs of the main theorems use 76 lemmas, spanning 1017 lines total.</li>
</ul>
<p>Our lemmas are largely reusable and we’ll add them to the
Mi-Cho-Coq framework to help future contract certifications, by us or by others.</p>
<h1 id="moving-forward"><a class="toclink" href="#moving-forward">Moving forward</a></h1>
<p>At Nomadic Labs we are proud to have participated in the development and the verification of Dexter; the first smart-contract for exchanging tokens on the Tezos blockchain. </p>
<p>This work:</p>
<ol>
<li>improves the security of a key component of the Tezos blockchain, </li>
<li>has improved the open-source tooling for verifying smart contracts in Coq, </li>
<li>is a long-term investment in (and commitment to) future maintainability and security, and</li>
<li>defines more clearly what is assured, and what is not yet assured, of Dexter’s behaviour. </li>
</ol>
<p>Extending Mi-Cho-Coq is active R&D at Nomadic Labs, and we see potential applications beyond functional smart-contract specifications.</p>
<p>For example: the Trail of Bits audit of earlier versions of Dexter revealed <a href="https://github.com/trailofbits/publications/blob/master/reviews/dexter.pdf">exploits related to inter-contract communication</a> (which have been fixed), of a type which could not have been detected with Mi-Cho-Coq in its current
form. Improving this requires a finer modeling of Tezos’ smart-contract
execution model. Extending Mi-Cho-Coq in this direction, and proving
such properties on contracts such as Dexter, is future work
for the smart-contract verification team at Nomadic Labs.</p>
<p>Also, to further increase confidence in Dexter’s soundness, we envisage an equivalence proof
between Dexter and Uniswap. They work on
similar principles, and Uniswap has been extensively studied for
security bugs, so a proof that Dexter is equivalent (in a suitable formal sense) to Uniswap, would extend
the trust already held for Uniswap to Dexter.</p>
<h1 id="acknowledgments"><a class="toclink" href="#acknowledgments">Acknowledgments</a></h1>
<p>Thanks to <a href="https://camlcase.io/">camlCase</a> for their help during the verification effort, and to <a href="https://www.trailofbits.com/">Trail of Bits</a> for
discussing the finer points of their audit. We are very happy to have worked with them. </p>
<!-- proof of individual entrypoints -->
<!-- proof of main -->
<!-- informal specification -->
<!-- formal specification -->
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>Technically, Dexter can interact with any contract that
satisfies the <span class="caps">FA1</span>.2 specification. However, security issues may
arise if Dexter is integrated with a token contract that, whilst
satisfying <span class="caps">FA1</span>.2, is malicious. Dexter’s <a href="https://gitlab.com/camlcase-dev/dexter/-/blob/develop/docs/token-integration.md">Token integration
checklist</a>
specify a set of conditions that the integrated token must satisfy
to ensure safe operation. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>To be more precise: when applying a block, the Tezos protocol
inputs the list of operations contained therein and takes actions
accordingly. Suppose the next operation is a contract call, so
the protocol executes it. The return value may now be a list of
new contracts to call and though we’d like the protocol to execute
this list, we have not implemented this in Mi-Cho-Coq. We have
only implemented the function <code>interp</code> that is called during block
application. This function only executes one contract call and
then stops. Consequently, we cannot handle a contract call that
triggers other contract calls. Nor do we model balance or storage
updates. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>This count does not include the Dexter contract script, even
though it is also present in the development as a Coq
string. <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:4">
<p>Nomadic Labs is unaffiliated with USDtez or tzBTC. <a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Emmy+ in the partial synchrony model2020-10-09T12:00:00+02:002020-10-09T12:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2020-10-09:/emmy-in-the-partial-synchrony-model.html<p>We’d like to expand on our <a href="https://research-development.nomadic-labs.com/analysis-of-emmy.html">previous blog post analysing Emmy<sup>+</sup></a>.</p>
<p><strong>Note</strong>: Thanks to input from Bruno Blanchet
(Inria). The interested reader can experiment with <a href="https://gitlab.com/nomadic-labs/emmyplus-experiments/-/tree/partial_synchrony">our code used in the analysis</a>.
As in the <a href="/analysis-of-emmy.html">previous analysis</a>, we do not present any security proofs.</p>
<h2 id="to-recap"><a class="toclink" href="#to-recap">To recap</a></h2>
<p>Emmy<sup>+</sup> is the new Tezos consensus algorithm, adopted in October 2019.
Emmy<sup>+</sup> improved on the original Emmy (see our post on <a href="/emmy-an-improved-consensus-algorithm.html">Emmy<sup>+</sup>: an improved consensus algorithm</a>).<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup></p>
<p>In our previous <a href="/analysis-of-emmy.html">analysis of Emmy<sup>+</sup></a>
we assumed a perfect network that delivers blocks and
operations instantly to all participants. That is, using the terminology
below: we assumed a synchronous model with zero message delivery
time. This simplifying assumption was shared with the original
analyses of both <a href="https://eprint.iacr.org/2014/765.pdf">Bitcoin</a> and <a href="https://eprint.iacr.org/2016/889.pdf">Ouroboros</a>.</p>
<p>Now we will refine our previous assumption to something
more realistic: the <a href="https://groups.csail.mit.edu/tds/papers/Lynch/jacm88.pdf">partial
synchrony</a>
network model.</p>
<p><a href="https://decentralizedthoughts.github.io/2019-06-01-2019-5-31-models/">In brief</a>: <span class="math">\(\newcommand\mdelta{{{\Delta}}} \newcommand{\mtype}[1]{{\mathtt{#1}}} \newcommand\ie{\mathit{ie}} \newcommand\dpp{\mathit{dp}} \newcommand\de{\mathit{de}} \newcommand\edelay{\mathit{emmy\_delay}} \newcommand\edelaydelta{\edelay_{\Delta}}\)</span>
<a name="item1"></a></p>
<ol>
<li>In a <strong>synchronous</strong> model, message delivery delay is bounded by a known bound <span class="math">\(\mdelta\)</span>.
A message sent at time <span class="math">\(t\)</span> will be delivered by time <span class="math">\(t+\mdelta\)</span>.</li>
<li>In an <strong>asynchronous</strong> network model, message delivery time is unbounded (but each message must eventually be delivered).</li>
<li>In a <strong>partially synchronous</strong> model, a global bound <span class="math">\(\mdelta\)</span> exists, but its value is unknown.<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup></li>
</ol>
<p>Note: There are <a href="https://decentralizedthoughts.github.io/2019-09-13-flavours-of-partial-synchrony">two equivalent formulations</a> of the partially synchronous model: <em>unknown latency</em> (as above), and the <em>global stabilization time</em> (<span class="caps">GST</span>) flavor. We will briefly discuss <span class="caps">GST</span> <a href="#partial-synchrony-in-the-gst-flavor">towards the end of the post</a>.</p>
<h2 id="minimal-block-delay-in-partial-synchrony"><a class="toclink" href="#minimal-block-delay-in-partial-synchrony">Minimal block delay in partial synchrony</a></h2>
<p>So this blog post profiles Emmy<sup>+</sup> in a partially synchronous network.</p>
<p>Emmy<sup>+</sup> defines a <strong>minimal block delay function</strong> as a function of the current block’s priority <span class="math">\(p\)</span>, and the number of endorsements <span class="math">\(e\)</span> included in the current block:</p>
<div class="math">$$\edelay(p, e) = 60 + \dpp \cdot p + \de\cdot \max(0, \ie - e) .
$$</div>
<p>Above</p>
<ul>
<li><span class="math">\(\dpp\)</span> stands for “delay per priority” (that is, per missed baking slot),</li>
<li><span class="math">\(\de\)</span> stands for “delay per (missed) endorsement (slot)”,</li>
<li><span class="math">\(\ie\)</span> stands for number of initial endorsements, and</li>
<li>the constant 60 represents the minimal time between the blocks.</li>
</ul>
<p>The <a href="/analysis-of-emmy.html">previous analysis of Emmy<sup>+</sup></a> effectively set <span class="math">\(\mdelta\)</span> to 0, so the same function could be used for the analysis. Now to account for partial synchrony, we elaborate <span class="math">\(\edelay\)</span> to capture message delays. We consider a worst case scenario, in which the network maximally favors the attacker, thus:</p>
<ul>
<li>the honest baker’s messages get delayed by the maximum possible <span class="math">\(\mdelta\)</span> while</li>
<li>the attacker’s messages arrive instantly.</li>
</ul>
<p>In symbols, the minimal block delay function <span class="math">\(\edelaydelta(p, e)\)</span> becomes</p>
<ul>
<li><span class="math">\(\edelaydelta(p, e) = \min(\max(2\mdelta, \edelay(p, e)),
\max(\mdelta, \edelay(p, 0)))\)</span>
for blocks baked by an honest baker, and remains as</li>
<li><span class="math">\(\edelaydelta(p, e) = \edelay(p, e)\)</span> for blocks baked by the attacker.</li>
</ul>
<p>The new delay function can be understood as follows:</p>
<ul>
<li>once a block is baked, it takes <span class="math">\(\mdelta\)</span> time units
for the other honest bakers to receive it and</li>
<li>it takes another <span class="math">\(\mdelta\)</span> time units to gather the endorsements for
the new block to be baked, consequently</li>
<li>the earliest “baking time” is
<div class="math">$$
\max(2\mdelta, \edelay(p, e)).
$$</div>
</li>
<li>If the baker prefers to bake its block without endorsements, then the
earliest “baking time” is
<div class="math">$$
\max(\mdelta, \edelay(p, 0)) .
$$</div>
</li>
</ul>
<p><a name="obs1"></a>
<strong>Observation</strong> 1: If <span class="math">\(\mdelta\leq 30s\)</span> then
</p>
<div class="math">$$
\edelaydelta(p, e) = \edelay(p, e) .
$$</div>
<p>
So, the previous analysis also holds in a synchronous network model, provided <span class="math">\(\mdelta\leq 30s\)</span>.</p>
<h2 id="fork-probabilities"><a class="toclink" href="#fork-probabilities">Fork probabilities</a></h2>
<p>As in our <a href="/analysis-of-emmy.html">previous analysis</a>, we consider a
<a href="https://lamport.azurewebsites.net/pubs/byz.pdf">Byzantine attacker</a>
— so the attacker is cost-insensitive and could double sign<sup id="fnref:3"><a class="footnote-ref" href="#fn:3">3</a></sup>. The
attacker wants to undo a transaction by</p>
<ol>
<li>baking an alternative chain which does not include the transaction, and</li>
<li>revealing this chain at a later point, thus creating a “fork”.</li>
</ol>
<p>The attacker’s chain must become faster than the “honest” chain to be adopted as the new chain. So basically, we must compute the probability that the
timestamp of the head of the dishonest chain is less than the
timestamp of the head of the honest chain. This difference
can be expressed in terms of differences between minimal block delays, and this is
where <span class="math">\(\mdelta\)</span> enters the analysis. See
<a href="on-defending-against-malicious-reorgs-in-tezos-proof-of-stake.html#on-the-methodology">here</a>
for details.</p>
<p>In what follows, we analyze the impact of <span class="math">\(\mdelta\)</span> on fork
probabilities. We recall that the <a href="/analysis-of-emmy.html">previous analysis</a> considered two main scenarios</p>
<ul>
<li><strong>forks starting now</strong> and</li>
<li><strong>forks starting in the past</strong>,</li>
</ul>
<p>which we will reconsider below.</p>
<p>In both cases, we ask when a transaction can be considered “final”, that is:</p>
<blockquote>
<p>How many <em>confirmations</em> (blocks on top of the block containing the relevant transaction) must we observe to be <strong>reasonably sure</strong> the transaction will remain in the chain?</p>
</blockquote>
<p>Here, <em>reasonably sure</em> means “with probability smaller than some
reasonable threshold”, which we quantify as <span class="math">\(\small 10^{-8}\)</span>
(which puts our expectation of being wrong about a block being final at <a href="/analysis-of-emmy.html#preliminaries">roughly once every two centuries</a>). We call
<strong>confirmation number</strong> the expected number of confirmations for a
block to be considered final.</p>
<h3 id="forks-starting-now"><a class="toclink" href="#forks-starting-now">Forks starting now</a></h3>
<p>Anybody with a small child will know that the contents of their fork may be recorded on the tablecloth, and there may only be a certain time period after which this becomes indelible.</p>
<p>In a similar vein we can ask of Tezos:</p>
<blockquote>
<p>Given that my transaction belongs to the current blockchain head,
how many confirmations do I need to see
to be reasonably sure that my transaction is final?</p>
</blockquote>
<p>We addressed this in our <a href="/analysis-of-emmy.html#forks-starting-now">previous
analysis</a>: we
provided 7, 16, 67 as confirmation numbers for .2, .3,
respectively .4 attacker stake fraction and <span class="math">\(\mdelta= 0s\)</span>. By
<a href="#obs1">Observation 1</a>, the same holds also
for <span class="math">\(\mdelta \leq 30s\)</span>.</p>
<p>We redo our computations with the new delay function <span class="math">\(\edelaydelta\)</span>
for <span class="math">\(\mdelta = 60s\)</span>. We see that the confirmation numbers only
increase more visibly for .4 attacker stake.</p>
<p><a name="fig1"></a>
<img alt="image" src="images/emmyplus_partial_sync/forks_starting_now_as_fct_of_delta_60.png"></p>
<p>However, for larger <span class="math">\(\mdelta\)</span>, the data suggests that we can no longer provide a
decent criterion for high attacker stake fractions. For instance, for <span class="math">\(\mdelta =
120s\)</span>, already for an attacker stake fraction of 0.3, the
confirmation number exceeds 300.</p>
<p><a name="fig2"></a>
<img alt="image" src="images/emmyplus_partial_sync/forks_starting_now_as_fct_of_delta.png"></p>
<h3 id="forks-started-in-the-past"><a class="toclink" href="#forks-started-in-the-past">Forks started in the past</a></h3>
<p>Now suppose we were absent from the room and returned to find our child surrounded by spilled food.
After which length of absence do we know there is no hope of cleaning up?</p>
<p>Likewise we can ask of Tezos:</p>
<blockquote>
<p>I have seen <span class="math">\(n\)</span> confirmations since my transaction was included in a
block, <span class="math">\(d\)</span> time ago; is my transaction final?</p>
</blockquote>
<p>We addressed this in our <a href="/analysis-of-emmy.html#forks-started-in-the-past">previous
analysis</a>: we
considered the case where the <strong>accumulated delay</strong> is <span class="math">\(d-60
\cdot n\)</span>, or in words: the delay of the current head with respect to
the earliest time the current head could ideally have been baked,
counting from the block where the transaction was included.</p>
<p>A more relevant case is that of a healthy network, that is, one with 0
accumulated delay (which is the usual case on the mainnet) and with
<span class="math">\(\mdelta\leq 30s\)</span>. Notably, in this case, the values for <span class="math">\(n\)</span> are 3, 5,
and 12 for .2, .3, respectively .4 attacker stake fraction. These
numbers are obtained with the same implementation as the one used in
<a href="/analysis-of-emmy.html#forks-started-in-the-past">previous analysis</a>,
even though they do not appear as such there.</p>
<!-- Before redoing the computations for different $\mdelta$ values, let us
first make a remark: -->
<p><strong>Remark</strong>: the fact that, at the current moment, the chances that
there exists already a (covert) feasible fork (started at least <span class="math">\(n\)</span>
healthy blocks ago, with <span class="math">\(n\)</span> as above) are tiny, does not mean that
<span class="math">\(n\)</span> is a suitable confirmation number, because if
network delays become large henceforth, then the attacker could (in
theory) take advantage of this to overtake the honest chain. Put
another way: if network delays are possible, then we cannot consider a
block <span class="math">\(n\)</span> levels deep to be final <em>even if</em> <span class="math">\(n\)</span> is large and
everything has gone well so far.</p>
<p>We now set about formalizing this remark. We first consider a
variant as follows:</p>
<blockquote>
<p>Assume the attacker started its chain <span class="math">\(n\)</span> blocks ago.
What is the least <span class="math">\(n\)</span> such that the probability that the
attacker has a faster chain if revealed <strong>now</strong>, is smaller than our
security threshold <span class="math">\(\small10^{-8}\)</span>?</p>
</blockquote>
<p>This variant differs from the question implicit in the Remark above in that the variant
disregards the future. This means that, since we assume 0 accumulated
delay, the computation of the probability of forks only depends on the
attacker’s stake, and not on <span class="math">\(\mdelta\)</span>, because we assume that the
network “works” in its favor. We obtain the following values for <span class="math">\(n\)</span>,
given the following stake fractions <span class="math">\(f\)</span> (<span class="math">\(f\)</span> varied in increments of <span class="math">\(0.005\)</span>):</p>
<ul>
<li><span class="math">\(n=2\)</span> for <span class="math">\(0.1 \leq f \leq 0.295\)</span>,</li>
<li><span class="math">\(n=3\)</span> for <span class="math">\(0.3 \leq f \leq 0.435\)</span>,</li>
<li><span class="math">\(n=4\)</span> for <span class="math">\(0.44 \leq f \leq 0.495\)</span>.</li>
</ul>
<p>These are small numbers, and we can show they remain small even for higher
security thresholds. However, as the plot below suggests, as soon as we consider
the probability of forks of <strong>any</strong> length (therefore also considering the
future), the expected confirmation numbers grow rapidly with <span class="math">\(\mdelta\)</span>.</p>
<p><a name="fig3"></a>
<img alt="image" src="images/emmyplus_partial_sync/forks_started_in_the_past_as_fct_of_n_delta.png"></p>
<p>Finally, the following plot takes a different perspective, showing how fork probabilities grow with <span class="math">\(\mdelta\)</span>, for a fixed number <span class="math">\(n\)</span> of seen blocks and when
the chain was healthy during these last <span class="math">\(n\)</span> blocks. We take the same values of
for <span class="math">\(n\)</span>, namely 3, 5, and 12 for .2, .3, respectively .4 attacker, as obtained
at the beginning of the section.</p>
<p><a name="fig4"></a>
<img alt="image" src="images/emmyplus_partial_sync/forks_started_in_the_past_as_fct_of_delta.png"></p>
<p>We can see from the plot that fork probabilities grow rapidly with
<span class="math">\(\mdelta\)</span> to a point where an attacker can build a fork at each level.</p>
<h3 id="partial-synchrony-in-the-gst-flavor"><a class="toclink" href="#partial-synchrony-in-the-gst-flavor">Partial synchrony in the <span class="caps">GST</span> flavor</a></h3>
<p>In the <a href="https://decentralizedthoughts.github.io/2019-09-13-flavours-of-partial-synchrony/"><strong><span class="caps">GST</span>
flavor</strong></a>
of the partially synchronous model, there is first a period when the asynchrony
assumption holds, followed by a period when the synchrony assumption holds. The
end of the asynchrony period is called the <strong>global stabilization time</strong> (<span class="caps">GST</span>),
which is assumed to be unknown. However, the upper bound <span class="math">\(\mdelta\)</span> on the
message delay in the synchrony period is known a priori, and therefore can be
used by the consensus algorithm.</p>
<p>All the above analysis is valid in the case when forks start after <span class="caps">GST</span>. However,
when forks start before <span class="caps">GST</span>, we have no guarantees at all regarding finality:
the blocks and operations of the honest parties may be delayed arbitrarily, and
when this happens the attacker’s chain can take over the honest chain. Indeed,
suppose that there are already <span class="math">\(n\)</span> confirmations, but that the next block is
delayed for longer than the time needed by the adversary to build a fork of
length <span class="math">\(n+1\)</span>: the attacker’s chain will thus become the main chain.</p>
<p>Note that we can see the case when are before <span class="caps">GST</span> as a particular one of the <span class="caps">UL</span>
flavor where <span class="math">\(\mdelta\)</span> tends towards infinity.</p>
<h2 id="conclusions"><a class="toclink" href="#conclusions">Conclusions</a></h2>
<p>Nakamoto-style consensus algorithms
such as those implemented by the <a href="https://eprint.iacr.org/2014/765.pdf">Bitcoin</a>,
<a href="https://eprint.iacr.org/2016/889.pdf">Ouroboros</a>, and <a href="https://eprint.iacr.org/2016/919.pdf">Snow
White</a> protocols rely on <a
href="item1">synchrony</a> and the knowledge of <span class="math">\(\mdelta\)</span> (which is
implicit in the algorithm design) in order to bound the expected number of
confirmations needed for a transaction to be considered as “final” — above, we called this a <em>confirmation number</em>.</p>
<p>This post provides empirical evidence that the same
observation holds for Emmy<sup>+</sup>: we need synchrony to guarantee low confirmation numbers and fork probabilities.</p>
<p>The plots above contain curves for both large and small <span class="math">\(\mdelta\)</span>, and illustrate that:</p>
<ul>
<li>for reasonable network delays, when <span class="math">\(\mdelta \leq 60s\)</span>, the confirmation number increases only slightly, and mainly for higher attacker
stake fractions, with respect to the case when <span class="math">\(\mdelta \leq 30s\)</span> (see Figures <a href="#fig1">1</a>, <a href="#fig2">2</a>, <a href="#fig3">3</a>, and <a href="#fig4">4</a>).</li>
<li>for higher network delays, when <span class="math">\(\mdelta > 60s\)</span>, the expected number of confirmations degrades quickly, even for smaller attacker stake fractions (see Figures <a href="#fig2">2</a>, <a href="#fig3">3</a>, and <a href="#fig4">4</a>).</li>
</ul>
<p>Specifically, the data suggests that to get decent expected confirmation numbers, it would be sufficient that the upper bound on message delays be less than the minimal time between blocks.
In practice, this is extremely realistic: the least time between consecutive blocks is <a href="#delay">60s</a>, and this is a very generous practical upper bound on message delays.<sup id="fnref:4"><a class="footnote-ref" href="#fn:4">4</a></sup> However, if the minimal time between blocks does not represent an upper bound on message delays, then the risk of forks increases significantly (see Figure <a href="#figD">D</a>).</p>
<h3 id="next-steps"><a class="toclink" href="#next-steps">Next steps</a></h3>
<p>We recall two advantages of the classical <span class="caps">BFT</span>-like algorithms:</p>
<ul>
<li>They have <strong>deterministic finality</strong>: the finality of a transaction is ensured
with probability 1 instead of being ensured with high probability.</li>
<li>They provide the finality guarantee even when the network is asynchronous.</li>
</ul>
<p>The price to pay for this is that we must make a stronger assumption
that the attacker holds <em>at most</em> a third of the total stake. This
trade-off may be worthwhile and we are experimenting accordingly with
versions of the <a href="https://arxiv.org/abs/1807.04938">Tendermint
algorithm</a>, including with our
in-house adaptation:
<a href="https://arxiv.org/abs/2001.11965">Tenderbake</a>. That’s a topic for
another blog post, so stay tuned!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>The post is from June 2019. The changes went into the Babylon proposal, which was injected in July. The proposal was accepted in October 2019. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>In practice the difference between synchrony and partial synchrony is this: In a synchronous network <span class="math">\(\mdelta\)</span> is known so a receiver knows that if a message was not received by <span class="math">\(t+\mdelta\)</span>, then it was never sent. In partial synchrony, the bound <span class="math">\(\mdelta\)</span> exists but the receiver does not know it, so the receiver cannot time out in the same way. What is usually done is to make successively increasing <em>guesses</em>: in this round of the algorithm we wait 10sec, in the next round 20sec, etc. Eventually there will be a round when we wait longer than <span class="math">\(\mdelta\)</span>.
Note that <span class="math">\(\mdelta\)</span> need not be a secret; the issue is just whether the <em>algorithm</em> may depend on it having a fixed, hardwired value. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p><span class="dquo">“</span>Double signing” is a consequence of “double baking” or “double
endorsing”, both of which are forbidden by the algorithm. A baker double bakes when
she produces two different blocks at the same level. A baker double endorses
when she endorses two different blocks at the same level. We recall that
both baking and endorsing require the baker to sign the relevant block. <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:4">
<p>The 60s upper bound is extremely realistic on an average day, but perhaps one day a year it may not be. That’s one of the problems with assuming synchrony. <a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>On “Defending Against Malicious Reorgs in Tezos Proof-of-Stake”2020-09-09T12:00:00+02:002020-09-09T12:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2020-09-09:/on-defending-against-malicious-reorgs-in-tezos-proof-of-stake.html<p><strong>Note</strong>: We would like to acknowledge Bruno Blanchet (Inria) for his feedback. The interested
reader can experiment with <a href="https://gitlab.com/nomadic-labs/emmyplus-experiments/-/tree/response_to_aft_paper">our code used in the
analysis</a>.</p>
<p>We are happy to note a new paper on Emmy<sup>+</sup>: <a href="https://arxiv.org/abs/2009.05413">“Defending Against
Malicious Reorgs in Tezos Proof-of-Stake”</a> by
Michael Neuder, Daniel J. Moroz, Rithvik Rao, and David C. Parkes in the <a href="https://aft.acm.org/"><span class="caps">ACM</span>
Advances in Financial Technologies 2020</a> conference.</p>
<p>We are delighted that Tezos and Emmy<sup>+</sup> are stimulating independent research
activity, and we hope to see much more of this in the future.</p>
<p>The paper mainly studies the rate at which malicious forks can occur, developing on ideas
put forward in our <a href="/analysis-of-emmy.html">blog post on analyzing
Emmy<sup>+</sup></a>.
Where the paper’s analysis overlaps with our analysis, its conclusions agree with ours overall.</p>
<p>We also note a nice feature of the paper: a chain health metric (<a href="https://arxiv.org/abs/2009.05413">Section
5</a>), which allows users to “identify potentially
vulnerable chain states and take extra care with accepting large transactions”.
This is close to our notion of <em>accumulated/observed delay</em>: the higher the
accumulated delay, the less healthy is the chain and the more confirmations a
user should wait for: see the <a href="/analysis-of-emmy.html#forks-started-in-the-past">“Forks started in the past”
section</a>
part of our Emmy<sup>+</sup> analysis.</p>
<p>We’d now like to discuss:</p>
<ol>
<li>
<p>an alternative concrete choice of parameter values for Emmy<sup>+</sup> considered in
the paper, and</p>
</li>
<li>
<p>the methodology of our original analysis.</p>
</li>
</ol>
<h3 id="the-concrete-choice-of-parameters"><a class="toclink" href="#the-concrete-choice-of-parameters">The concrete choice of parameters</a></h3>
<p>The paper considers an alternative set of parameter values for Emmy<sup>+</sup>
(<a href="https://arxiv.org/abs/2009.05413">Subsection 4.3</a>).
For this choice, the rate of <a href="/analysis-of-emmy.html#forks-starting-now">“forks starting now”</a> is smaller when considering high
attacker stake fractions (e.g. 0.4 or more).</p>
<p>The exploration of the parameter space is interesting, but we chose our
parameter values as we did for quite practical reasons: we believe that in the
real world, attackers will probably attack from a position of relatively smaller
stake (less than 0.4, say), and for reasons discussed in detail <a href="#comparing-the-two-approaches">below
(“Comparing the two approaches”)</a> we find the
<a href="/analysis-of-emmy.html#forks-started-in-the-past">“forks started in the past”
scenario</a> more appropriate for
defending Tezos against malicious forks.</p>
<h3 id="the-methodology"><a class="toclink" href="#the-methodology">The methodology</a></h3>
<p>The paper’s approach is mostly simulation-based (Monte Carlo with importance
sampling): exact probability calculations are given only for short fork lengths
because the method used is enumeration based and thus it “quickly becomes
computationally intractable” (<a href="https://arxiv.org/abs/2009.05413">Subsection 2.6 <span class="amp">&</span> Appendix A,
B</a>).</p>
<p>The data in <a href="/analysis-of-emmy.html">our previous blog post</a> was entirely
generated by an efficient implementation
of an exact method for computing probabilities. We describe it in the more
technical part of the post <a href="#on-the-methodology">below (“On the methodology”)</a>
where we show that calculating probabilities is, in fact, <em>linear</em> in fork
length, and <em>quadratic</em> in the number of samples used to approximate the
relevant probability distributions. More samples gives the analysis more
precision, but we can still compute usefully precise probabilities for a large
range of attacker stake fractions — even as high as 0.4-0.45, depending on the experiment.</p>
<p>Now, for the benefit of the more technically-minded reader, let’s take a closer
look at the details:</p>
<h2 id="on-choosing-the-constants"><a class="toclink" href="#on-choosing-the-constants">On choosing the constants</a></h2>
<h3 id="nomadic-labs-approach"><a class="toclink" href="#nomadic-labs-approach">Nomadic Labs’ approach</a></h3>
<p>We recall that the parameters <code>ie</code>, <code>dp</code>, <code>de</code> were chosen while designing
Emmy<sup>+</sup>. That is, they were chosen for the Babylon update. The reward
policy has changed in Carthage to take into account <a href="/a-new-reward-formula-for-carthage.html">various profitability
criteria of selfish
bakers</a>,
in particular criteria that take inflation into account.</p>
<p>The methodology used in <a href="/analysis-of-emmy.html">our previous blog post</a> was as follows:</p>
<ul>
<li>
<p>We tried to minimize both the feasibility of malicious forks, and of “block
stealing”, that is, the <em>feasibility</em> of selfish baking. However, we did not
take the <em>profitability</em> of selfish baking into account, the idea being that
the reward policy can deal with this aspect. We believe that finding both the
parameters <code>ie</code>, <code>dp</code>, <code>de</code>, <em>and</em> also the reward policy in a single step is
too complicated: the search space is too big, and it is even harder to obtain
an intuition on the dependencies between the parameters.</p>
</li>
<li>
<p>We first observed that it is sufficient to search for appropriate values of
<code>ie</code> and <code>dp/de</code>.</p>
</li>
<li>
<p>We took the range of <code>ie</code> as <code>[16, 30]</code> (with a step of <code>2</code>) and that of
<code>dp/de</code> <code>[1,20]</code> and, for stake fractions from <code>0.2</code> to<code>0.45</code> (with a step of
<code>0.05</code>), we computed the rate of malicious forks and the rate of block stealing.</p>
</li>
<li>
<p>For malicious forks, we considered the <a href="/analysis-of-emmy.html#forks-started-in-the-past">“forks started in the past”
scenario</a>, assuming that a
fork has started <code>5</code> blocks in the past and the observed delay during these <code>5</code>
blocks was equivalent to <code>16</code> missing endorsements.</p>
</li>
<li>
<p>Among these values for <code>ie</code> and <code>dp/de</code>, we chose <code>ie = 24</code> and <code>dp/de = 5</code>
as, at a visual inspection of the plots, they seemed to be a good compromise
for the two conflicting minimization problems: minimizing the feasibility of
malicious forks and minimizing the feasibility of block stealing.</p>
</li>
<li>
<p>We then chose <code>dp = 40</code> as it seemed to be a good interval to reduce the
waiting time between blocks (with respect to the previous value used in Emmy,
namely 75), while also being large enough to allow blocks and operations to be
timely delivered.</p>
</li>
</ul>
<h3 id="the-papers-approach"><a class="toclink" href="#the-papers-approach">The paper’s approach</a></h3>
<p><a href="https://arxiv.org/abs/2009.05413">The paper</a> proposes an objective function (in
Section 4.2, equation (14)) which weighs both the rate of malicious forks, and
of selfish baking, for two given forks length, namely 20 for malicious forks and
3 for selfish baking. One would then look for parameter values that minimize
this objective function.</p>
<p>The paper sets parameter values of <code>ie=15</code>, <code>dp=8</code>, and <code>de=5</code>.
For an attacker stake fraction of <code>0.4</code> these values give a weighted probability, as
given by the objective function, that is 2.5 times smaller (so better) than the
current parameter values for Emmy<sup>+</sup>, namely, <code>ie=24</code>, <code>dp=40</code>, <code>de=8</code>.</p>
<h3 id="comparing-the-two-approaches"><a class="toclink" href="#comparing-the-two-approaches">Comparing the two approaches</a></h3>
<p>First, the paper argues about the optimality of the new parameters in a quite
particular setting: for high attacker stake fractions, and for specific and
somewhat arbitrary forks lengths. However, these choices represent only a few
points in a much larger space of possible combinations. It is unclear whether
one would reach the same conclusion when looking at the whole space.</p>
<p>Second, the paper considers the <a href="/analysis-of-emmy.html#forks-starting-now">“forks starting now”
scenario</a> and not the <a href="/analysis-of-emmy.html#forks-started-in-the-past">“forks started
in the past” scenario</a>, which
is the one a user is ultimately interested in: ideally, a user would not blindly
wait for a number of confirmations (as provided by the <a href="/analysis-of-emmy.html#forks-starting-now">“forks starting
now”</a> scenario), instead she would
look at the health of the chain since the time her transaction was included and
would take this information into account (as in the <a href="/analysis-of-emmy.html#forks-started-in-the-past">“forks started in the past”
scenario</a>).</p>
<p>Note that using <a href="/analysis-of-emmy.html#forks-started-in-the-past">“forks started in the past”
scenario</a> means that we
compare a rather healthy chain (the case that is most often observed) to a chain
made by the adversary. In contrast, if we look at <a href="/analysis-of-emmy.html#forks-starting-now">“forks starting now”
scenario</a>, the honest chain is
typically less healthy, since the adversary does not provide endorsements
and does not bake on the honest chain. In other words, when looking at the
<a href="/analysis-of-emmy.html#forks-started-in-the-past">“forks started in the past”
scenario</a>, we would optimize
for the desired case, while when looking at <a href="/analysis-of-emmy.html#forks-starting-now">“forks starting now”
scenario</a>, we would optimize for the
worst case. So both approaches have their strengths.</p>
<p>Third, we would be cautious about choosing a small <code>dp</code> value, because this
intuitively means that if the block at priority 0 is missing, then the bakers at
the following priorities (say 1, 2, 3) have little time to see each other’s
blocks, so we might get that half of the network is on the block at priority 1
and half on the block at priority 2 (assuming that all the endorsements have
already arrived). However, one could still achieve the same proposed <code>dp/de</code>
ratio of <code>8/5</code> with higher <code>dp</code> values, e.g. <code>dp=40</code> and <code>de=25</code>.</p>
<p>In what follows we revisit and improve parts of our previous analysis to
understand how the newly proposed parameters compare with the ones currently
used by Emmy<sup>+</sup>. For completeness, we provide <a href="https://gitlab.com/nomadic-labs/emmyplus-experiments/-/tree/response_to_aft_paper">the code used for our
experiments</a>.</p>
<p>For the sake of comparison, we replot forks started now/in the past with <code>ie=15</code>
and <code>dp/de=8/5</code>.</p>
<p><img alt="Image" src="images/aft20_response/now_NL_15_8_5.png"></p>
<p>So for <a href="/analysis-of-emmy.html#forks-starting-now">“forks starting now”</a>, the
choice proposed in the paper is better.</p>
<p>However, the choice <code>(ie,dp/de)=(24,5)</code> is slightly better for <a href="/analysis-of-emmy.html#forks-started-in-the-past">“forks started
in the past” scenario</a> for
observed delay 0.</p>
<p><img alt="Image" src="images/aft20_response/past_NL_15_8_5.png"></p>
<p>In an attempt to be more systematic, inspired by the paper, we plotted a few
additional <a href="#appendix">contour maps</a>.</p>
<h2 id="on-the-methodology"><a class="toclink" href="#on-the-methodology">On the methodology</a></h2>
<p>The main point of this section is to show that the complexity of our approach is
<em>linear</em> in the length of the fork. Said otherwise, an analytical approach does
not need to be exponential in the length of the fork, as suggested in the paper.</p>
<p>Our approach is to compute the probability distribution of the time difference
between the attacker’s chain and the honest chain, while assuming both chains
have equal length. By the “time of a chain” we mean the timestamp of the head of
the chain. Note that the attacker is able to produce a fork if and only if this
difference is smaller than 0.</p>
<p>The probability distribution is computed on a bounded sample space. Namely, we
fix lower and upper bounds on the timestamp difference. Concretely, we only
computed the probabilities that the difference is in the interval <code>[-m'/2,
m'/2]</code>, for some constant <code>m'</code>. Note that the higher <code>m'</code>, the higher the
precision of the analysis.</p>
<p>As showed in <a href="/analysis-of-emmy.html">the previous blog post</a>, it is the ratio
<code>dp/de</code> that matters in this analysis. For convenience, we have assumed that
<code>dp</code> is a multiple of <code>de</code>. Therefore we have that any possible timestamp
difference is a multiple of <code>de</code>. This means that we can represent the
probability distribution by an array of size <code>m</code> where <code>m'=m/de</code> and where an
index <code>i</code> corresponds to the difference <code>(i-m/2)*de</code>.</p>
<p>In our experiments, <code>m</code> was usually <code>2001</code>. This is sufficient, in that we do
not lose too much precision: we checked that the sum of all probabilities in the
bounded sample space is still <code>1</code> and not smaller (or at least not significantly smaller).</p>
<p>We illustrate in more detail how the probability distribution is computed for
the <a href="/analysis-of-emmy.html#forks-starting-now">“forks starting now”</a> case. The
other cases are similar.</p>
<p>The main idea is to compute the probability distribution inductively: given the
probability distribution for forks of length <code>n</code>, compute the probability
distribution for forks of length <code>n+1</code>. Then, at each iteration we check whether
we reached a fork length for which the probability that the attacker’s chain is
faster than the honest chain is smaller than the security parameter.
The pseudo-code of the main procedure is given below.</p>
<div class="highlight"><pre><span></span><code><span class="nv">one_level_pd</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">init</span><span class="p">(</span><span class="nv">first_level</span><span class="o">=</span><span class="nv">False</span><span class="p">)</span>
<span class="nv">pd</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">init</span><span class="p">(</span><span class="nv">first_level</span><span class="o">=</span><span class="nv">True</span><span class="p">)</span>
<span class="nv">win_p</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="mi">0</span>
<span class="k">for</span><span class="nv">k_len</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="mi">1</span>
<span class="k">while</span><span class="w"> </span><span class="nv">win_p</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mf">1e-8</span><span class="o">:</span>
<span class="w"> </span><span class="nv">pd</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nf">sum_pds</span><span class="p">(</span><span class="nv">pd</span><span class="p">,</span><span class="w"> </span><span class="nv">one_level_pd</span><span class="p">)</span>
<span class="w"> </span><span class="nv">win_p</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nv">pd</span><span class="p">[</span><span class="o">:</span><span class="nv">m</span><span class="o">/</span><span class="mi">2</span><span class="p">]</span><span class="o">.</span><span class="nf">sum</span><span class="p">()</span>
<span class="w"> </span><span class="k">for</span><span class="nv">k_len</span><span class="o">++</span>
<span class="nv">return</span><span class="w"> </span><span class="k">for</span><span class="nv">k_len</span>
</code></pre></div>
<p>The <code>init</code> procedure computes the probability distribution of the difference
between the delay of the attacker’s block and of the honest block at some
arbitrary level. Recall that if the level is that of the first block of the
fork, then the attacker can also include the endorsements of the honest parties,
while otherwise it can only include its own endorsements. This distinction is
captured with the <code>first_level</code> argument.</p>
<p>The <code>init</code> procedure iterates through the <code>m</code> possible differences and the 33
possible number of endorsements the attacker might have. For each combination we
determine the priorities the attacker and the honest parties should have in
order to obtain the corresponding difference. Based on this information, we
compute the probability to obtain the given difference.</p>
<p>The <code>sum_dps</code> procedure basically computes the convolution of two probability
distributions. In our case, <code>sum_pds</code> computes the probability distribution of
the difference between chains of length <code>n+1</code> as the convolution of the
probability distribution between chains of length <code>n</code> and the probability
distribution of the difference between chains of length 1. Concretely, <code>sum_pds</code>
goes through all pairs <code>(i,j)</code> of indexes in <code>pd</code> and respectively
<code>one_block_pd</code>, such that the differences <code>d_k</code> and <code>d_1</code> corresponding to
indexes <code>i</code> and <code>j</code> add up to a difference <code>d_{k+1} = d_k + d_1</code> in the <code>[-m'/2,
m'/2]</code> interval. In that case the new probability associated with the difference
<code>d_{k+1}</code> is updated to <code>p_{k+1} := p_{k+1} + p_k * p_1</code>, where <code>p_k</code> is the
probability associated with <code>d_k</code> in <code>pd</code> and <code>p_1</code> is the probability
associated with <code>d_1</code> in <code>one_level_pd</code>. (<code>p_{k+1}</code> is initially set to <code>0</code>.)</p>
<p>The main procedure is then a simple while loop in which at each iteration we
compute the probability that the attacker’s chain is faster than the honest
chain. This condition is equivalent to saying that the difference between the
two chains is negative. This is why the corresponding probability is obtain as
the sum of the probabilities in the first half of the <code>pd</code> array.</p>
<p>From the explanations above, it is easy to see that the time complexity of a
call to <code>init</code> is <code>O(m)</code> and the time complexity of a call to <code>sum_pds</code> is
<code>O(m^2)</code>. We conclude that the complexity of this approach is <em>linear</em> in the
length of the fork and <em>quadratic</em> in <code>m</code>.</p>
<h3 id="a-visual-explanation"><a class="toclink" href="#a-visual-explanation">A visual explanation</a></h3>
<p>We plot the arrays <code>pd</code> for <code>f=.3</code> and <code>m=1001</code>. We intentionally consider a
rather small value of <code>m</code> because it is small enough that we get errors bigger
than 1e-9, where by “error” we mean the difference between 1 and the sum of all
probabilities in <code>pd</code>.</p>
<p><img alt="Image" src="images/aft20_response/chain_diff_30_1001_default_params.png"></p>
<p>For a value <code>x0</code> on the x-axis, the corresponding point <code>y0</code> in the
curve for a given iteration represents that the probability that the
difference between the attacker chain and the honest chain is
<code>x0*de</code>. Recall that the attacker chain is faster (so attacker’s fork
“wins”) if and only if this difference is smaller than 0. Therefore,
for a given iteration <code>i</code> the area under the curve <code>iteri</code> at left of
0 approximates the probability that an attacker wins for any
difference with a fork of length <code>i+1</code>, while the total area should be
1. We note however that since the curves go a bit more to the right at
each iteration (as the expected time difference between the chains
increases) computing the total sum up to <code>m/2</code> loses data (the
truncated part of the shifted curves). To see this more clearly, we
zoom in.</p>
<p><img alt="Image" src="images/aft20_response/chain_diff_30_1001_default_params_zoom.png"></p>
<p>For <code>m=2001</code>, we no longer get an error thus this sample size is sufficiently
large to not lose information.</p>
<p><img alt="Image" src="images/aft20_response/chain_diff_30_2001_default_params.png"></p>
<p>Though for <code>m=1001</code> the results are imprecise with respect to the
total sum, the resulting probabilities are the same as for
<code>m=2001</code>. This is because the truncation only affects visibly the
right-hand side while the probability that the attacker wins is on the
left-hand side of 0: the lost data on the right-hand side does not
have a visible impact. Still, for safety we updated the code so that
if the sum is not close to 1, an error is raised suggesting the user
to use a larger <code>m</code>.</p>
<p>For completeness, we add a version of the previous plot, where the
y-axis is in log scale.</p>
<p><img alt="Image" src="images/aft20_response/chain_diff_30_2001_default_params_log.png"></p>
<h3 id="appendix"><a class="toclink" href="#appendix">Appendix</a></h3>
<p>Instead of heatmaps, we plot contour maps, in log scale, as we find they suit our purpose better in that we can see clearly the delimitations between different security thresholds (represented by different coloured regions). As the security threshold we considered so far is <span class="math">\(10^{-8}\)</span>, the region of interest in each plot is the one with the colour corresponding to -8 in the colour bar.</p>
<p>In the plots below, the z-axis is the probabilities for forks starting now, resp. in the past.</p>
<p>For forks starting now, we consider the fork lengths 1, 2, 3, and 6 and 20 and exhaustively explore the following ranges for <code>ie</code> and <code>dpde</code>: <code>ie</code> ranges from 14 to 32 and <code>dpde</code> ranges from 1 to 20.</p>
<p><img alt="Image" src="images/aft20_response/forks_now_params_spectrum_1_2_3_6_20.png"></p>
<p>The plots suggest that smaller <code>dpde</code> values are better.</p>
<p>This is not necessarily the case for forks started in the past. We
consider stake fractions .2, resp. .3 and we vary the accumulated
delay for 3 seen blocks for .2, resp. 5 and 10 seen blocks for .3. (We
recall that 3, resp. 5 were the expected confirmation numbers for .2,
resp. .3 when the accumulated delay is 0). Precisely, what we denote
by <code>odr</code> in the plots represents the accumulated (or observed) delay
divided by <code>de</code> and <code>odr</code> ranges over {0, 16, 32, 64, 128}.</p>
<p><img alt="Image" src="images/aft20_response/forks_past_params_spectrum_20_3_30_5_10_0_16_32_64_128.png"></p>
<p>The above plots suggest that higher <code>ie</code> values are better. Moreover,
we find that it is more important to choose higher <code>ie</code> values despite
the observation that smaller <code>ie</code> values are “better” when considering
forks starting now. This is because for forks started in the past, a
small <code>ie</code> may cause high fork probabilities. For instance: if the
attacker stake fraction is .2, the number of seen blocks is 3, <code>odr</code>
is 32, and <code>dpde</code> is around 5, then the probability of a fork can go
up to <span class="math">\(10^{-2}\)</span> for small <code>ie</code> values such as 15. In contrast, for the
same setup but with high <code>ie</code> values such as 30, the probability of a
fork remains smaller than our security threshold of <span class="math">\(10^{-8}\)</span>.</p>
<p>And finally: we had not previously considered smaller <code>ie</code> values (as
this was against the “raison d’être” of Emmy). We checked this with
some experiments as illustrated in the plots below, in which <code>ie</code>
ranges from 0 to 14. These plots demonstrate that for such small <code>ie</code>
values, the fork probabilities are high (compared to the above plots
for forks starting now), and that higher <code>ie</code> are better.</p>
<p><img alt="Image" src="images/aft20_response/forks_now_params_spectrum_1_2_3_6_20.png"></p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Delphi: official release2020-09-03T13:01:00+02:002020-09-03T13:01:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2020-09-03:/delphi-official-release.html<p>Summary:</p>
<h1 id="delphi"><a class="toclink" href="#delphi">Delphi</a></h1>
<p><em>This is a joint post from Nomadic Labs, Metastate and Gabriel Alfour.</em></p>
<p>We’re happy to announce that we have compiled a Tezos protocol proposal we’re dubbing “Delphi”, with hash <code>PsDELPH1Kxsxt8f9eWbxQeRxkjfbxoqM52jvs5Y5fBxWWh4ifpo</code>. Delphi contains a number of small bug fixes, but, more importantly, it makes substantial improvements to …</p><p>Summary:</p>
<h1 id="delphi"><a class="toclink" href="#delphi">Delphi</a></h1>
<p><em>This is a joint post from Nomadic Labs, Metastate and Gabriel Alfour.</em></p>
<p>We’re happy to announce that we have compiled a Tezos protocol proposal we’re dubbing “Delphi”, with hash <code>PsDELPH1Kxsxt8f9eWbxQeRxkjfbxoqM52jvs5Y5fBxWWh4ifpo</code>. Delphi contains a number of small bug fixes, but, more importantly, it makes substantial improvements to the performance of the Michelson interpreter and to the gas model, and thus dramatically improves gas costs. In addition, it reduces storage costs by a factor of 4 to reflect improvements in the underlying storage layer.</p>
<p>Many of you have probably been playing with the Dalphanet test network, which includes support for <a href="https://research-development.nomadic-labs.com/sapling-integration-in-tezos-tech-preview.html">Sapling</a>, <a href="https://medium.com/metastatedev/stateful-baking-accounts-d483a63a173c">baking accounts</a>, and more. The teams contributing to that code have projected that testing and tuning will be done in several months.</p>
<p>During the <a href="https://youtu.be/e5bxiqxGI5U">online town hall meeting</a> on August 19th, it was asked whether an interim protocol proposal could be made that introduces gas improvements sooner, as this would be of substantial help to people developing smart contracts on Tezos.</p>
<p>The motivation for such an interim proposal is straightforward. The size and complexity of smart contracts is limited by gas constraints, and so people attempting to build contracts with rich functionality have needed improvements to those constraints for some time. Thus, such improvements are crucial to enable novel applications on Tezos that target areas like DeFi (“Decentralized Finance”), collectibles, and gaming. Luckily, in August, we finalized some long-standing work on improving the performance of the Michelson type checker and interpreter, and on refining the cost model, thus mitigating the gas problem.</p>
<p>We carefully considered the idea of an interim proposal, and realized that the suggestion was practical. A small protocol proposal is simpler to create and evaluate than a large one, and there is currently an opportunity to inject an interim protocol proposal focused on improving gas constraints without significantly changing the expected proposal date for the code in Dalphanet. Because it is a very limited set of changes, it was feasible to assemble a proposal in time.</p>
<p>The finished proposal, which we have named “Delphi”, incorporates the subset of changes to the Tezos protocol in Dalphanet that considerably improve performance and relax gas constraints.</p>
<p>Here are some examples of the performance gains you can expect to see in Delphi:</p>
<ul>
<li>A block may now include 3.5 times more simple tz<em> to tz</em> ops</li>
<li>A block may now include 4 times more <span class="caps">FA2</span> transfers</li>
<li>A contract may perform 10 times more internal calls</li>
</ul>
<p>The precise improvement you can expect will vary significantly depending on the specific case, but importantly, the benefits will be most visible in large and complex contracts involving multiple calls to other contracts and substantial computation.</p>
<p>(A complete list of updates in the proposal has been made available in the <a href="https://research-development.nomadic-labs.com/delphi-changelog.html">previous post</a>.)</p>
<p>As for Sapling, baking accounts, and all the rest: the code for those is feature complete and available for members of the community to play with in the Dalphanet test network. We encourage you to try out Dalphanet; the more testing it gets, the better the final proposal will be.</p>
<p>After the end of the amendment process for Delphi, we intend to introduce a new proposal based on the features currently in Dalphanet that are judged to be ready at that time. If Delphi is adopted, this next proposal (which may have a name starting with the letter “E”) could be injected and enter the Tezos amendment process in early December.</p>Delphi: changelog2020-09-03T13:00:00+02:002020-09-03T13:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2020-09-03:/delphi-changelog.html<p>Summary:</p>
<h1 id="007-delphi-changelog"><a class="toclink" href="#007-delphi-changelog">007 Delphi Changelog</a></h1>
<h2 id="getting-the-source-code-of-the-proposal"><a class="toclink" href="#getting-the-source-code-of-the-proposal">Getting the source code of the proposal</a></h2>
<p>The source code of this proposal is available in this tar archive: <a href="https://research-development.nomadic-labs.com/files/delphi_007_PsDELPH1.tar">https://research-development.nomadic-labs.com/files/delphi_007_PsDELPH1.tar</a>.</p>
<p>To compute the hash, you can use the <code>tezos-protocol-compiler</code> as follows. The result should be <code>PsDELPH1Kxsxt8f9eWbxQeRxkjfbxoqM52jvs5Y5fBxWWh4ifpo</code>.</p>
<div class="highlight"><pre><span></span><code>mkdir proposal && cd proposal …</code></pre></div><p>Summary:</p>
<h1 id="007-delphi-changelog"><a class="toclink" href="#007-delphi-changelog">007 Delphi Changelog</a></h1>
<h2 id="getting-the-source-code-of-the-proposal"><a class="toclink" href="#getting-the-source-code-of-the-proposal">Getting the source code of the proposal</a></h2>
<p>The source code of this proposal is available in this tar archive: <a href="https://research-development.nomadic-labs.com/files/delphi_007_PsDELPH1.tar">https://research-development.nomadic-labs.com/files/delphi_007_PsDELPH1.tar</a>.</p>
<p>To compute the hash, you can use the <code>tezos-protocol-compiler</code> as follows. The result should be <code>PsDELPH1Kxsxt8f9eWbxQeRxkjfbxoqM52jvs5Y5fBxWWh4ifpo</code>.</p>
<div class="highlight"><pre><span></span><code>mkdir proposal && cd proposal
curl https://research-development.nomadic-labs.com/files/delphi_007_PsDELPH1.tar | tar -x
tezos-protocol-compiler -hash-only delphi_007_PsDELPH1
</code></pre></div>
<p>If you want to directly investigate how Delphi differs from the current protocol, you can use <code>diff</code> against folder <code>proto_006_PsCARTHA</code> of the mainnet source code. All the changes should be the ones detailed in the changes section.</p>
<h2 id="injecting-the-proposal"><a class="toclink" href="#injecting-the-proposal">Injecting the proposal</a></h2>
<p>Any active baker can inject the proposal by simply downloading the code, injecting it in a running mainnet node with <code>tezos-admin-client</code> and using the <code>submit</code> command of <code>tezos-client</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-admin-client<span class="w"> </span>inject<span class="w"> </span>protocol<span class="w"> </span>delphi_007_PsDELPH1
Injected<span class="w"> </span>protocol<span class="w"> </span>PsDELPH1Kxsxt8f9eWbxQeRxkjfbxoqM52jvs5Y5fBxWWh4ifpo<span class="w"> </span>successfully
$<span class="w"> </span>tezos-client<span class="w"> </span>submit<span class="w"> </span>proposals<span class="w"> </span><span class="k">for</span><span class="w"> </span><delegate><span class="w"> </span>PsDELPH1Kxsxt8f9eWbxQeRxkjfbxoqM52jvs5Y5fBxWWh4ifpo
</code></pre></div>
<div class="toc">
<ul>
<li><a href="#007-delphi-changelog">007 Delphi Changelog</a><ul>
<li><a href="#getting-the-source-code-of-the-proposal">Getting the source code of the proposal</a></li>
<li><a href="#injecting-the-proposal">Injecting the proposal</a></li>
<li><a href="#performance-improvements">Performance Improvements</a></li>
<li><a href="#gas-changes">Gas Changes</a><ul>
<li><a href="#change-in-format-of-receipts-related-to-gas">Change in format of receipts related to gas</a></li>
</ul>
</li>
<li><a href="#lowered-storage-costs">Lowered storage costs</a></li>
<li><a href="#bug-fixes">Bug fixes</a><ul>
<li><a href="#compatibility-with-32-bit-platforms">Compatibility with 32-bit platforms</a></li>
<li><a href="#breaking-change-michelson-annotations"><span class="caps">BREAKING</span> <span class="caps">CHANGE</span>: Michelson annotations</a></li>
<li><a href="#miscellanous-bug-fixes">Miscellanous bug fixes</a></li>
</ul>
</li>
<li><a href="#general-code-cleanup-and-refactoring">General code cleanup and refactoring</a><ul>
<li><a href="#removal-of-lwt-when-possible">Removal of Lwt when possible</a></li>
<li><a href="#miscellanous-code-improvements-refactors">Miscellanous code improvements <span class="amp">&</span> refactors</a></li>
</ul>
</li>
<li><a href="#safety-of-michelson">Safety of Michelson</a><ul>
<li><a href="#explicit-limitations-in-the-michelson-typechecker-and-interpreter">Explicit limitations in the Michelson typechecker and interpreter</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</div>
<h2 id="performance-improvements"><a class="toclink" href="#performance-improvements">Performance Improvements</a></h2>
<p>Some effort was directed to streamline the Michelson interpreter and improve its performance. This mostly consisted in factoring gas consumption, simplifying logging and removing some calls to <code>Lwt</code> (cf. the dedicated section on that last point).</p>
<p>Patches:</p>
<ul>
<li>217b4fd025 Proto/Michelson: abstract logging facility</li>
<li>7504b35b94 Proto/Michelson: make lists carry their size and type of elements</li>
<li>c744b2e36e Proto/Michelson: hoist gas consumption in Slice_{string,bytes}</li>
<li>9854ddc12a Proto/Michelson: hoist Gas.consume out of interp. pattern matching</li>
</ul>
<h2 id="gas-changes"><a class="toclink" href="#gas-changes">Gas Changes</a></h2>
<p>Gas accounting has seen a significant update. Even though gas limits are the same as in Carthage, the amount of computation per unit of gas should see a significant increase in Delphi.</p>
<p>More precisely, the following parts of the gas subsystem have been updated.
- Following the optimization work on the interpreter, gas costs for all instructions have been recomputed.
- The cost model for IOs has been updated and should account better for performances of modern storage hardware.
- The typechecking system has an entirely new cost model.
- Finally, the base gas cost of manager operations has been reduced from 10000 to 1000 units of gas.</p>
<p>Overall, users should see significant gas cost reductions accross the board.</p>
<p>Patches:</p>
<ul>
<li>8747f0a987 Proto/Michelson: Use the right cost function for SET_DELEGATE instr</li>
<li>6825b14cc0 Protocol/Michelson: fix gas consumption in Loop exit case</li>
<li>267ddd55d9 Protocol/gas: fix scaling of gas accounting for number of writes</li>
<li>f1397bffb6 Proto/gas: make internal gas visible in interpreter traces</li>
<li>1a874fb990 Proto/gas: rescale gas by 2^7, making internal_gas always 0</li>
<li>866fa99e71 Proto/gas: remove internal_gas</li>
<li>3eab3718a4 Proto/Michelson: properly carbonate extract_big_map_updates and collect_big_maps</li>
<li>a5baf0ed59 Proto/Michelson: Remove gas cost of ‘map_to_list’ and ‘set_to_list’ conversions</li>
<li>1ae30c0932 Protocol/Migration: scale gas limit constants in the context by 128</li>
<li>76918bf6fa Proto/Gas: remove intermediary cost record</li>
<li>3e11d74d91 Proto/Gas: factor calls to scaling function</li>
<li>07f89a7a9e Proto/Gas: inline calls to scale</li>
<li>c67f0dd61c Proto/Gas: perform rescaling from x128 to x1000</li>
<li>786bdb360f Proto/Gas: arithmetic optimization (remove multiplication by 1)</li>
<li>6c611b948d Protocol/Migration: scale gas limit constants in the context by 1000</li>
<li>5dfb79a814 Revert “Protocol/Migration: scale gas limit constants in the context by 1000”</li>
<li>fbccb2390a Revert “Vendors/flextesa: scale gas limit constants in the context by 1000”</li>
<li>f3201231f2 Revert “Protocol/Migration: scale gas limit constants in the context by 128”</li>
<li>b50d6f870a Revert “Vendors/flextesa: scale gas limit constants in the context by 128”</li>
<li>27bb7aeb2d Proto/gas: set 1 milligas = 1 atomic_step_cost</li>
<li>7e50a74755 Proto/Gas: add new gas cost functions in dedicated module</li>
<li>097e79f626 Proto/Gas: inject new cost functions in <code>Gas.cost</code>, expose in .mli</li>
<li>78a2e8069c Proto/Gas: plug new costs on the interpreter</li>
<li>02b5fc516c Proto/Gas: fix gas for Concat_string <span class="amp">&</span> Concat_bytes</li>
<li>e00ec5eff8 Proto/Gas: unplug 006 interpreter gas</li>
<li>1838b7da21 Proto/Gas: introduce Storage_costs module</li>
<li>a21c1ed287 Proto/Gas: use Storage_costs in carbonated storage functors</li>
<li>3281de90c2 Proto/Gas: introduce Contract.get_balance_carbonated</li>
<li>2e1a9d88e3 Proto/Gas: adding encoding/decoding cost constants</li>
<li>68b4c2c37d Proto/Gas: injecting Gas_limit_repr.cost into Alpha_context.Gas.cost</li>
<li>33a0d8f015 Proto/Gas: expose 007 typechecking costs</li>
<li>be9c3159d8 Proto/Gas: Adapt translator to 007 typechecking costs</li>
<li>bd205ad65e Proto/Gas: expose 007 unparsing costs</li>
<li>eb784b35be Proto/Gas: adapt translator to 007 unparsing costs</li>
<li>9b478326fd Proto/Gas: add cost functions and helpers for strip_locations</li>
<li>b574bff205 Proto/Gas: carbonate calls to Micheline.strip_locations</li>
<li>6e6d59f578 Proto/Gas: Unplug 006 translator gas</li>
<li>9ae9b3c771 Proto/Gas: prettify constants</li>
<li>987c4f93a7 Proto/Gas: remove useless <code>let</code> bindings in cost functions</li>
<li>4ffc37e769 Proto/tests: add basic unit test for cost functions</li>
<li>5ef9a9aae1 Proto/Gas: reduce gas cost of manager op</li>
</ul>
<h3 id="change-in-format-of-receipts-related-to-gas"><a class="toclink" href="#change-in-format-of-receipts-related-to-gas">Change in format of receipts related to gas</a></h3>
<p>The protocol now internally handles <em>milligas</em> but exposes gas to the user.
- Protocol constants such as per-block and per-operation gas limits are <strong>still specified in gas</strong>.
- On the input side of the protocol, the <code>gas_limit</code> field in manager operations is <strong>still specified in gas</strong>, as before.
- On the output side, receipts contain both the <code>consumed_gas</code> (as before) and a new field <code>consumed_milligas</code>. The relationship between this two fields is the following:</p>
<div class="math">$$
consumed\_gas = \lceil consumed\_milligas / 1000 \rceil
$$</div>
<p>In other terms, <code>consumed_gas</code>, provided for backwards compatibility, is the rounded-up version of <code>consumed_milligas</code>. Users wanting to have the most accurate description of the gas consumption of their transfers should consider using <code>consumed_milligas</code>.</p>
<p>To make understanding the change easier, here is a small example illustrating a transfer in sandboxed mode to a toy <code>minimal.tz</code> contract:</p>
<div class="highlight"><pre><span></span><code>{ parameter int ;
storage unit ;
code { CDR ; NIL operation; PAIR } }
</code></pre></div>
<p>Let us originate this contract.</p>
<div class="highlight"><pre><span></span><code>tezos-client -l originate contract mini transferring 0 from bootstrap1 running minimal.tz --burn-cap 0.07375 --init 'Unit'
</code></pre></div>
<p>During simulation, the <code>preapply</code> <span class="caps">RPC</span> is given the following operation:</p>
<div class="highlight"><pre><span></span><code>{ "protocol": "ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK",
"branch": "BMSAKmQD2Q2Ghk2jtHMGpx5Zww98oLsFdh4XW1863TizUjyfY8S",
"contents":
[ { "kind": "origination",
"source": ..., "fee": "434", "counter": "1",
"gas_limit": "1690",
"storage_limit": "315",
"balance": "0",
"script": ... }
</code></pre></div>
<p>The <code>gas_limit</code>, equal to <code>1690</code> in this example, is labelled in <strong>gas</strong>.
The outcome of the simulation is the following receipt:</p>
<div class="highlight"><pre><span></span><code>{ "kind": "origination",
"source": ..., "fee": "434",
"counter": "1",
"gas_limit": "1690",
"storage_limit": "315", "balance": "0",
"script": ...,
"metadata":
{ "balance_updates": ...,
"operation_result":
{ "status": "applied", "big_map_diff": [],
"balance_updates": ...,
"originated_contracts": ...,
"consumed_gas": "1590",
"consumed_milligas": "1589562",
"storage_size": "38",
"paid_storage_size_diff": "38" } } }
</code></pre></div>
<p>The metadata contains both:
- the field <code>consumed_milligas</code>, corresponding to the <strong>exact</strong> amount of gas consumed, labelled in <strong>milligas</strong>;
- the field <code>consumed_gas</code>, obtained by ceiling the milligas value as described above, labelled in <strong>gas</strong>.</p>
<p>Note that the <code>tezos-client</code> binary prints gas, not milligas. However it displays three decimals:</p>
<div class="highlight"><pre><span></span><code><span class="nt">This</span><span class="w"> </span><span class="nt">sequence</span><span class="w"> </span><span class="nt">of</span><span class="w"> </span><span class="nt">operations</span><span class="w"> </span><span class="nt">was</span><span class="w"> </span><span class="nt">run</span><span class="o">:</span>
<span class="w"> </span><span class="nt">Manager</span><span class="w"> </span><span class="nt">signed</span><span class="w"> </span><span class="nt">operations</span><span class="o">:</span>
<span class="w"> </span><span class="nt">From</span><span class="o">:</span><span class="w"> </span><span class="nt">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span>
<span class="w"> </span><span class="nt">Fee</span><span class="w"> </span><span class="nt">to</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">baker</span><span class="o">:</span><span class="w"> </span><span class="nt">ꜩ0</span><span class="p">.</span><span class="nc">000434</span>
<span class="w"> </span><span class="nt">Expected</span><span class="w"> </span><span class="nt">counter</span><span class="o">:</span><span class="w"> </span><span class="o">...</span>
<span class="w"> </span><span class="nt">Gas</span><span class="w"> </span><span class="nt">limit</span><span class="o">:</span><span class="w"> </span><span class="nt">1690</span>
<span class="w"> </span><span class="nt">Storage</span><span class="w"> </span><span class="nt">limit</span><span class="o">:</span><span class="w"> </span><span class="nt">315</span><span class="w"> </span><span class="nt">bytes</span>
<span class="w"> </span><span class="nt">Balance</span><span class="w"> </span><span class="nt">updates</span><span class="o">:</span>
<span class="w"> </span><span class="nt">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span><span class="w"> </span><span class="o">...........</span><span class="w"> </span><span class="nt">-ꜩ0</span><span class="p">.</span><span class="nc">000434</span>
<span class="w"> </span><span class="nt">fees</span><span class="o">(</span><span class="nt">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span><span class="o">,</span><span class="nt">0</span><span class="o">)</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="o">+</span><span class="nt">ꜩ0</span><span class="p">.</span><span class="nc">000434</span>
<span class="w"> </span><span class="nt">Origination</span><span class="o">:</span>
<span class="w"> </span><span class="nt">From</span><span class="o">:</span><span class="w"> </span><span class="nt">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span>
<span class="w"> </span><span class="nt">Credit</span><span class="o">:</span><span class="w"> </span><span class="nt">ꜩ0</span>
<span class="w"> </span><span class="nt">Script</span><span class="o">:</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">parameter</span><span class="w"> </span><span class="err">int</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="err">storage</span><span class="w"> </span><span class="err">unit</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="err">code</span><span class="w"> </span><span class="err">{</span><span class="w"> </span><span class="err">CDR</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="err">NIL</span><span class="w"> </span><span class="err">operation</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="err">PAIR</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="err">}</span>
<span class="w"> </span><span class="nt">Initial</span><span class="w"> </span><span class="nt">storage</span><span class="o">:</span><span class="w"> </span><span class="nt">Unit</span>
<span class="w"> </span><span class="nt">No</span><span class="w"> </span><span class="nt">delegate</span><span class="w"> </span><span class="nt">for</span><span class="w"> </span><span class="nt">this</span><span class="w"> </span><span class="nt">contract</span>
<span class="w"> </span><span class="nt">This</span><span class="w"> </span><span class="nt">origination</span><span class="w"> </span><span class="nt">was</span><span class="w"> </span><span class="nt">successfully</span><span class="w"> </span><span class="nt">applied</span>
<span class="w"> </span><span class="nt">Originated</span><span class="w"> </span><span class="nt">contracts</span><span class="o">:</span>
<span class="w"> </span><span class="nt">KT1HUgv4V8RnEhzNXUxWSqU2PGZ6MgRh34n3</span>
<span class="w"> </span><span class="nt">Storage</span><span class="w"> </span><span class="nt">size</span><span class="o">:</span><span class="w"> </span><span class="nt">38</span><span class="w"> </span><span class="nt">bytes</span>
<span class="w"> </span><span class="nt">Paid</span><span class="w"> </span><span class="nt">storage</span><span class="w"> </span><span class="nt">size</span><span class="w"> </span><span class="nt">diff</span><span class="o">:</span><span class="w"> </span><span class="nt">38</span><span class="w"> </span><span class="nt">bytes</span>
<span class="w"> </span><span class="nt">Consumed</span><span class="w"> </span><span class="nt">gas</span><span class="o">:</span><span class="w"> </span><span class="nt">1589</span><span class="p">.</span><span class="nc">562</span>
<span class="w"> </span><span class="nt">Balance</span><span class="w"> </span><span class="nt">updates</span><span class="o">:</span>
<span class="w"> </span><span class="nt">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="nt">-ꜩ0</span><span class="p">.</span><span class="nc">0095</span>
<span class="w"> </span><span class="nt">tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx</span><span class="w"> </span><span class="o">...</span><span class="w"> </span><span class="nt">-ꜩ0</span><span class="p">.</span><span class="nc">06425</span>
</code></pre></div>
<p>Patches:</p>
<ul>
<li>7868fa947b Proto/gas: introduce Fixed_point_repr module</li>
<li>c436869c49 Proto/tests: unit tests for Fixed_point_repr</li>
<li>5a4f408582 Proto/gas: edit TEZOS_PROTOCOL for Fixed_point, move Gas_limit_repr</li>
<li>e939017844 Proto/gas: use fixed-point computations for gas (<span class="caps">FIXME</span>)</li>
<li>6031a0dc8c Proto/client: use fixed-point computations for gas</li>
<li>7d9f76418d Proto/delegate: use fixed-point computations for gas</li>
<li>804b42cd86 Proto: restore format of receipts to a retrocompatible schema</li>
</ul>
<h2 id="lowered-storage-costs"><a class="toclink" href="#lowered-storage-costs">Lowered storage costs</a></h2>
<p>In Tezos, storing data in the state leads to a burn of tez proportional to the size of the storage increase. This happens when creating a new account, originationg a new smart contract or making the storage of a smart contract grow above its historical higher size. This is different from fees that are paid to the baker proportionally to the size of operations. Indeed, operations are transient (and can be forgotten by nodes in the lighweight rolling mode), while data in the state is replicated by all nodes and can be so forever.</p>
<p>In Delphi, the amount of tez burned to store data in the ledger’s state is decreased by a factor of 4, going from 1 tez to 0.25 tez for a kilobyte. The price to create a new account is thus lowered from 0.257 tez to 0.06425.</p>
<p>Patches:</p>
<ul>
<li>8f808e8317 proto/parameters: reduce cost_per_byte to 0.000250tz</li>
<li>ffaba729db proto/migration: set cost_per_byte to 0.000250tz</li>
</ul>
<h2 id="bug-fixes"><a class="toclink" href="#bug-fixes">Bug fixes</a></h2>
<h3 id="compatibility-with-32-bit-platforms"><a class="toclink" href="#compatibility-with-32-bit-platforms">Compatibility with 32-bit platforms</a></h3>
<p>Carthage has some parts that assume a 64-bit runtime. These modifications restore the compatibility with 32-bit systems, and in general make the code clearer and less dependent on the underlying word size.</p>
<p>Patches:</p>
<ul>
<li>da91297c0d Protocol/Michelson: avoid overflowing [Int32.to_int Int32.max_int]</li>
<li>377af3acb0 Proto/Michelson: Simplify parse_uint30</li>
<li>06c2f6f97b Protocol: Safer Int64.to_int</li>
<li>40f9a2c9a3 Storage: rename Int -> UInt16</li>
<li>ceb4ef33ba Storage: rename Int_index -> Int31_index</li>
<li>70c0aa4641 Proto/Gas: Zarith-ify some cost functions</li>
</ul>
<h3 id="breaking-change-michelson-annotations"><a class="toclink" href="#breaking-change-michelson-annotations"><span style="color:red"><span class="caps">BREAKING</span> <span class="caps">CHANGE</span></span>: Michelson annotations</a></h3>
<p>Field annotations in Michelson types were not properly checked and could contain invalid characters. In particular, it was possible to use a digit as the first meaningful character of a field annotation but only if it appeared in a type; a few contracts on Mainnet contain such numerals as first meaningful character. We have added the missing check and extended the syntax of Michelson annotations to allow digits. At the time of writing, all Mainnet and Carthagenet contracts successfully typecheck.</p>
<p><span style="color:red"><strong><span class="caps">BREAKING</span> <span class="caps">CHANGE</span></strong></span>: If a smart contract containing a non-numeral invalid character in a field annotation inside a type were to be originated before the activation of Delphi, such a contract would be locked by the activation.</p>
<p>Patches:</p>
<ul>
<li>ec1d992e5c Use plain algebraic types for the Michelson annotations</li>
<li>642bab2f97 Proto/Michelson: ensures all annotations are checked</li>
<li>0f12f628e9 Proto/Michelson: extend the set of allowed Michelson annotations</li>
<li>1b179aeb0b Proto/Michleson: make annotations on <code>Right</code> and <code>Elt</code> consistent</li>
</ul>
<h3 id="miscellanous-bug-fixes"><a class="toclink" href="#miscellanous-bug-fixes">Miscellanous bug fixes</a></h3>
<p>Patches:</p>
<ul>
<li>8c1dd8e53b Proto/Michelson: fix the arity check for chain_id</li>
<li>dde9e19d55 Proto/<span class="caps">RPC</span>: return all delegates if no flags were used</li>
<li>28b8181a8c Proto/Michelson: fix registration of error Invalid_syntactic_constant</li>
<li>7f329a1700 Proto/Michelson: add missing cases in typechecking error reporting</li>
<li>083e9c6f9b Proto: Add missing case for keyword namespace encoding</li>
<li>b1af688dfd Proto: for transfers with 0 tz, check if target exists</li>
<li>982dd6ec77 Proto: Update <code>max_revelations_per_block</code> to include anon ops</li>
<li>508b3ae5f4 Proto/Michelson: readable unparsing of chain ids</li>
<li>b03b8b0883 Proto: Fix double encoding of big_map_diff</li>
<li>ae0626d708 Proto: make signature check of operation packs more efficient</li>
</ul>
<h2 id="general-code-cleanup-and-refactoring"><a class="toclink" href="#general-code-cleanup-and-refactoring">General code cleanup and refactoring</a></h2>
<h3 id="removal-of-lwt-when-possible"><a class="toclink" href="#removal-of-lwt-when-possible">Removal of <code>Lwt</code> when possible</a></h3>
<p>A large batch of changes consists in mostly trivial modifications that drop parts of the code that don’t perform IOs out of the <span class="caps">IO</span> monad. This makes the code a bit less uniform, but helps with safety and future refactoring, and in some cases increases performance (in particular in the interpreter).</p>
<p>Patches:</p>
<ul>
<li>9e38b8fece Proto/Michelson: remove some Lwt wrappers</li>
<li>eda54e8ba3 Proto/Michelson: avoid Lwt.bind when logging</li>
<li>187d9d7cab Proto: bind -> to</li>
<li>b2f8aac20c List.map -> fold_left_s</li>
<li>85592c26e9 Proto: filter_map_s -> filter_s</li>
<li>14eda1e7a9 List.fold_left -> Error_monad.map</li>
<li>43306a4b89 Script: force_decode/bytes, get rid of Lwt.t</li>
<li>7b1199d76d Baking: minimal_(valid_)time, get rid of Lwt</li>
<li>c5a0f66bb0 Baking.earlier_predecessor_timestamp: get rid of Lwt</li>
<li>e112d592b2 Baking: baking/endorsing_reward, get rid of Lwt</li>
<li>622ac83eaf Baking/tests: baking/endorsing_reward, get rid of Lwt</li>
<li>287985b4c7 Baking.check_fitness_gap: get rid of Lwt</li>
<li>a9cdf7e3bc Contract_storage.fresh_contract_from_current_nonce: get rid of Lwt</li>
<li>aef7541f69 Delegate_services.required_endorsements/minimal_valid_time: get rid of Lwt</li>
<li>3854c15614 Fees_storage.origination_burn: get rid of Lwt</li>
<li>d71fe7c127 Operation.check_signature: get rid of Lwt</li>
<li>419d29b41b Proto: Lwt.return >>=? -> >>?=</li>
<li>36172136a2 Raw_context.add_fees/rewards/deposit: get rid of Lwt</li>
<li>f19e52bd86 Script_ir_annot: get rid of fail_unexpected_annot</li>
<li>b55bfc4673 Script_ir_translator: remove lots of Lwt in parse_instr</li>
<li>4381c2b266 Script_ir_translator.parse_data: remove lots of Lwt</li>
<li>509decd748 Script_ir_translator.parse_data: traced</li>
<li>8da99eff1a Script_ir_translator: less Lwt</li>
<li>407e460bce Script_ir_translator.collect_big_maps: get rid of Lwt</li>
<li>1c94741b2b Storage: less Lwt</li>
<li>30fad4cb40 Proto: fail_unless/when -> error_unless/when</li>
<li>01aeae86a3 Proto: use predefined ok constants</li>
<li>0b8964ad8f Proto: lift some Lwt.return</li>
</ul>
<h3 id="miscellanous-code-improvements-refactors"><a class="toclink" href="#miscellanous-code-improvements-refactors">Miscellanous code improvements <span class="amp">&</span> refactors</a></h3>
<p>A long series of patches consists in minor improvements to the code style and contents of comments or error messages. Part of this work is thanks to the automated transcription of the code of the protocol into Coq (as part of our formal verification effort).</p>
<p>Patches:</p>
<ul>
<li>88bc601fe9 Proto: Uniform variable names for context + comments clean-up</li>
<li>6d59b3f32a Michelson: exposes parsing without specifying storage</li>
<li>bef609828f Proto: expose parse_storage</li>
<li>abb3a3a48c Include ty_eq in merge_types</li>
<li>b1519b85e2 Express (comparable_)ty_eq in terms of merge_(comparable_)ty</li>
<li>d96d1cc2d1 Migration: remove leftovers Alpha_previous and Babylon_005</li>
<li>cd4bcec080 Proto/Michelson: remove dead code in parse_instr</li>
<li>74db352afd Remove mutual dependency between numeric types</li>
<li>60b8998900 Proto: remove unused exported vals</li>
<li>83aa3f49a5 Rewrite some when clauses for Coq</li>
<li>4b5307d995 Remove useless recursion</li>
<li>f34ec2a28d Proto: Fix formatting</li>
<li>99d7b81533 Proto: Move and export Michelson prim namespace function</li>
<li>29153f27ff Proto/Michelson: simplify the interpreter</li>
<li>36cbee348b Michelson: rename Left/Right as Cons_left/right</li>
<li>c6cba0db30 Michelson: simplify <span class="caps">GADT</span> matchings in script_ir_translator 1/4</li>
<li>e828029fb3 Michelson: simplify <span class="caps">GADT</span> matchings in script_ir_translator 2/4</li>
<li>0bff45c7f2 Michelson: simplify <span class="caps">GADT</span> matchings in script_ir_translator 3/4</li>
<li>edfae2bcf8 Michelson: simplify <span class="caps">GADT</span> matchings in script_ir_translator 4/4</li>
<li>cff2ab3b69 Proto/Michelson: extract_big_map_updates: aux</li>
<li>425f3eaaa1 Proto/Michelson: compute has_big_map only when needed</li>
<li>3b59480b92 Proto/Michelson: get rid of has_big_map flag and old function</li>
<li>dd34270a63 Proto/Michelson: factorize parse_storage_ty</li>
<li>8d17e58cc4 Proto/RPCs: use parse_packable/parameter_ty instead of parse_ty in services</li>
<li>09013da929 Proto/Contracts: big_map_diff/Copy: use inline record</li>
<li>084037dc9c Proto/Michelson: remove dead code</li>
<li>5e83fe9126 Proto/Michelson: simplify Big_map.fresh</li>
<li>0172dec928 Proto/Michelson: expose parse_ty for convenience to external tools</li>
<li>6c8e8a7f46 Proto/Michelson: expose unparse_code in Script_ir_translator</li>
<li>d793d30003 Proto_alpha: spell check</li>
<li>5807a28c44 Proto/coq-of-ocaml: Remove a polymorphic variant in raw_context</li>
<li>5da72db33f Proto: update the old operator to regular naming scheme</li>
<li>03ab590364 Protocol/coq-of-ocaml: name the signatures of the protocol</li>
<li>a06caece9f Protocol/coq-of-ocaml: avoid a name collision on encoding in the generated Coq</li>
<li>a25feb04f5 Use Option.value</li>
<li>12c7cf2f81 Protocol/coq-of-ocaml: rename of_seconds to prevent a collision of name</li>
<li>b8dd4fc1f9 Protocol/coq-of-ocaml: renaming of force_decode / force_bytes to prevent collision</li>
<li>c42b58c8c3 Protocol/coq-of-ocaml: rename consume and check_enough to prevent collisions</li>
<li>c4ac279d65 Protocol/coq-of-ocaml: renaming to compile alpha_context</li>
<li>d6ec7dd8c7 Protocol/coq-of-ocaml: lint the interpreter</li>
<li>da7d945551 Protocol/coq-of-ocaml: changes to compile storage.ml</li>
<li>75dad446a4 Protocol/coq-of-ocaml: add signature annotations</li>
<li>036c287157 Script_ir_translator.parse_data: rename error</li>
<li>b25d542877 Proto: fix typo in docstring</li>
<li>fe988f439c Proto: remove dead code</li>
</ul>
<h2 id="safety-of-michelson"><a class="toclink" href="#safety-of-michelson">Safety of Michelson</a></h2>
<h3 id="explicit-limitations-in-the-michelson-typechecker-and-interpreter"><a class="toclink" href="#explicit-limitations-in-the-michelson-typechecker-and-interpreter">Explicit limitations in the Michelson typechecker and interpreter</a></h3>
<p>Up until now, some features of Michelson were implicitly limited by gas. This series of changes add explicit limits that will help with safety, future refactoring, and debugging (more precise errors).</p>
<ul>
<li>Deep stack instructions <code>DIG</code>, <code>DUG</code>, <code>DROP</code>, and <code>DIP</code> are now bounded to stacks of less than 1024 elements</li>
<li>The interpreter (resp. typechecker) now have explicit recursion limits, setting a maximum depths for terms that can be interpreted (resp. typechecked).</li>
</ul>
<p>Both changes set limits to high enough values, and should thus be invisible to almost all contract authors. Reaching the limits is only likely to happen when debugging erroneous (such as non-terminating) code.</p>
<p>Patches:</p>
<ul>
<li>4c019e1004 Proto/Michelson: fix error message for <span class="caps">DIP</span> with wrong constant</li>
<li>7d0211b648 Michelson: fix number_of_generated_growing_types</li>
<li>6a7bbf17a4 Proto: add max stack depth</li>
<li>040abed403 Proto: normalize stackoverflows in typechecking and unparsing</li>
<li>95c31f963c Michelson: restrict deep stack instructions to 1023</li>
</ul>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Measuring test coverage2020-07-28T15:00:00+02:002020-07-28T15:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2020-07-28:/measuring-test-coverage.html<p>An in-depth look on how we measure test coverage of the OCaml implementation of Tezos.</p><p>As we have discussed earlier on this blog, notably in
<a href="https://research-development.nomadic-labs.com/catching-sneaky-regressions-with-pytest-regtest.html">the post on regression testing</a>,
testing is an important complement to formal
verification. A critical task when testing is to ensure a high degree
on test coverage. In this post we discuss how we measure test coverage
of the OCaml implementation of tezos using <a href="https://github.com/aantron/bisect_ppx"><code>bisect_ppx</code></a>,
and our work towards ensuring that the Michelson interpreter of the
node is fully covered by tests.</p>
<h2 id="test-coverage"><a class="toclink" href="#test-coverage">Test coverage</a></h2>
<p>Testing is simpler to set up than formal verification
but comes with a major caveat: it is incomplete. Each test run
verifies only <em>one execution path</em> — as opposed to formal methods
that can give guarantees on <em>all possible execution paths</em> of the
system under test.</p>
<p>The incompleteness of testing provokes a natural question in the
mind of the wary test engineer: to what extent do my tests cover the set
of behaviors of the tested system? Or, what parts of the code are
tested and what parts are not? Many measures have been proposed to
answer these and related questions, collected under the term of “test coverage”.</p>
<p>In this blog post, we focus on one such measure, namely program point
coverage. This consists of measuring the ratio of program points
(informally, a program point can be thought of as a location in the
source code) that have been visited during the test run. Other
measures include function coverage (which functions have been
visited?), edge coverage (which edges in the program’s control-flow
graph have been visited?) and condition coverage (which boolean
sub-expressions in guard-expressions have been evaluated to both true
and false).</p>
<p>However, full coverage <em>does not</em> imply
that all behaviors of the system has been explored, not for any of the measures above.
Notably, the
behavior of a program statement may change depending on the
program state when the statement is executed. But none of the above measures
quantify to what extent possible such program states are explored.
Instead, they are defined in terms of control-flow graph properties of
the tested executions. The takeaway is that full code coverage does
not guarantee full system correctness.</p>
<p>Nonetheless, there is still value in pursuing a high degree of test
coverage. First, it is an important proxy measure for the amount of
testing that has been performed. And second, because it detects code
that <em>has not been tested at all</em>. Such code may contain bugs
that are triggered when <a href="https://nakedsecurity.sophos.com/2014/02/24/anatomy-of-a-goto-fail-apples-ssl-bug-explained-plus-an-unofficial-patch/">executed in any program
state</a>. Systematically
testing code is a cheap insurance against such bugs.</p>
<p>In this section, we will detail our usage of the
<a href="https://github.com/aantron/bisect_ppx"><code>bisect_ppx</code></a> tool to measure
program point coverage in the Michelson interpreter by our test suite.</p>
<h3 id="the-bisect_ppx-code-coverage-tool"><a class="toclink" href="#the-bisect_ppx-code-coverage-tool">The <code>bisect_ppx</code> code coverage tool</a></h3>
<p><a href="https://github.com/aantron/bisect_ppx"><code>bisect_ppx</code></a> is a code coverage tool for OCaml and Reason programs.
A program that has been compiled with <code>bisect_ppx</code>
writes a log file with coverage data just before terminating.
A set of such log files can be converted into an <span class="caps">HTML</span> report
detailing and summarizing the set of program points that has been
visited during the execution.</p>
<p>For each instrumented <code>.ml</code> file that has been visited we obtain a
coverage measure (e.g. “73% of the program points were visited”), and
each file can be inspected in detail to understand what program points
have been visited and not.</p>
<figure style="text-align: center" class="image">
<img src="https://research-development.nomadic-labs.com/images/test_coverage/bisect_ppx_sample.gif" alt="Demonstration of bisect_ppx">
<figcaption style="margin-top: 5px">An example of using <tt>bisect_ppx</tt>, courtesy of
<a href="https://github.com/aantron/bisect_ppx">bisect_ppx</a>
</figcaption>
</figure>
<p><code>bisect_ppx</code> <em>instruments</em> the tested program by inserting code that
associates each program point with a counter that is incremented when
the program point is visited. Then, when compiled, the instrumented
program is linked with <code>bisect_ppx</code><span class="quo">‘</span>s run-time library. This library
registers an
<a href="https://caml.inria.fr/pub/docs/manual-ocaml/libref/Stdlib.html"><code>__atexit</code></a>
hook that is called at the end of the instrumented program’s execution.
This hook writes the program-point counters to a log file. At this
point, the developer can run their test suite to gather coverage data.
Finally, the <code>bisect-reporter-ppx</code> program included in
<code>bisect_ppx</code>, reads the log files and generates an <span class="caps">HTML</span> report.</p>
<h3 id="an-example-of-bisect_ppx-instrumentation"><a class="toclink" href="#an-example-of-bisect_ppx-instrumentation">An example of <code>bisect_ppx</code> instrumentation</a></h3>
<p>Consider the following test file for which we want to measure coverage:</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="n">add</span> <span class="n">n1</span> <span class="n">n2</span> <span class="o">=</span> <span class="n">n1</span> <span class="o">+</span> <span class="n">n2</span>
</code></pre></div>
<p>After running <code>bisect_ppx</code> we obtain the following, instrumented
source code:</p>
<div class="highlight"><pre><span></span><code><span class="k">module</span> <span class="nc">Bisect_visit___src___lib_addition___addition_lib___ml</span> <span class="o">=</span>
<span class="k">struct</span>
<span class="k">let</span> <span class="o">___</span><span class="n">bisect_visit___</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">point_definitions</span> <span class="o">=</span>
<span class="s2">"</span><span class="se">\132\149\166\190\000\000\000\004\000\000\000\002\000\000\000\005\000\000\000\005\144\160</span><span class="s2">P@"</span> <span class="k">in</span>
<span class="k">let</span> <span class="o">`</span><span class="nc">Staged</span> <span class="n">cb</span> <span class="o">=</span>
<span class="nn">Bisect</span><span class="p">.</span><span class="nn">Runtime</span><span class="p">.</span><span class="n">register_file</span> <span class="s2">"src/lib_addition/addition_lib.ml"</span>
<span class="o">~</span><span class="n">point_count</span><span class="o">:</span><span class="mi">1</span> <span class="o">~</span><span class="n">point_definitions</span> <span class="k">in</span>
<span class="n">cb</span>
<span class="k">end</span>
<span class="k">open</span> <span class="nc">Bisect_visit___src___lib_addition___addition_lib___ml</span>
<span class="k">let</span> <span class="n">add</span> <span class="n">n1</span> <span class="n">n2</span> <span class="o">=</span> <span class="o">___</span><span class="n">bisect_visit___</span> <span class="mi">0</span><span class="o">;</span> <span class="n">n1</span> <span class="o">+</span> <span class="n">n2</span>
</code></pre></div>
<p>We see that <code>bisect_ppx</code> has added a module
<code>Bisect_visit___src___lib_addition___addition_lib___ml</code> containing
a serialized definition of the program points in the source file, and
a function <code>___bisect_visit___</code> that increments a program point
counter. The function <code>add</code> has been associated to program point 0.
Hence, the <code>___bisect_visit___</code> function is called in the definition
of <code>add</code> with the argument 0. Consequently, each call to <code>add</code>
increments the counter for program point 0.</p>
<h3 id="test-coverage-for-the-michelson-interpreter"><a class="toclink" href="#test-coverage-for-the-michelson-interpreter">Test coverage for the Michelson interpreter</a></h3>
<p>We have used <code>bisect_ppx</code> to measure the test coverage of the
Michelson interpreter in the Tezos node, and to detect Michelson instructions that were
lacking test cases. We had two goals: first, to make sure that each
branch of the <code>step</code> function in the interpreter (corresponding roughly to each
Michelson instruction) is covered by at least one test. Second, to
make sure that each instruction has at least one dedicated test. Our
work on this topic can be followed in merge request
<a href="https://gitlab.com/tezos/tezos/-/merge_requests/1261">!1261</a>, that also
contains information on how to set up <code>bisect_ppx</code> for Tezos.</p>
<p>To measure progress on the first objective, we run <code>bisect_ppx</code> on the
full test suite of the Tezos node. For the second objective, we run it
using a test suite that targets individual Michelson instructions.</p>
<h4 id="baseline-results"><a class="toclink" href="#baseline-results">Baseline results</a></h4>
<!-- coverage at : 0478afc7 -->
<p>We take as our baseline commit <code>#0478afc7</code> of the Tezos node. This
version precedes our work towards full coverage, and for this reason has a low coverage.</p>
<!—
!— :: coverage reports ::
!—
!— baseline (Michelson instruction suite) : files/test_coverage/_coverage.20191003.1717.0478afc79.opcodes_sh
!— baseline (full) : files/test_coverage/_coverage.20191003.1735.0478afc79.contracts_sh
!— endline (Michelson instruction suite) : files/test_coverage/_coverage.20200708.1016.ec198d588b6c438246dac9dda30449d20510891f.contracts
!— endline (full) : files/test_coverage/_coverage.20200708.1035.ec198d588b6c438246dac9dda30449d20510891f.full
!—>
<p>The results using <code>bisect_ppx</code> for the Michelson instruction
test suite are available <a href="https://research-development.nomadic-labs.com/files/test_coverage/_coverage.20191003.1717.0478afc79.opcodes_sh/src/proto_alpha/lib_protocol/script_interpreter.ml.html">here</a>.
We see that the test suit for instructions covers 60.79% of the interpreter.
Several instructions are missing tests: instructions for big maps such as
<a href="https://research-development.nomadic-labs.com/files/test_coverage/_coverage.20191003.1717.0478afc79.opcodes_sh/src/proto_alpha/lib_protocol/script_interpreter.ml.html#L408"><code>EMPTY_BIG_MAP</code></a>,
<a href="https://research-development.nomadic-labs.com/files/test_coverage/_coverage.20191003.1717.0478afc79.opcodes_sh/src/proto_alpha/lib_protocol/script_interpreter.ml.html#L411"><code>MEM</code></a> and
<a href="https://research-development.nomadic-labs.com/files/test_coverage/_coverage.20191003.1717.0478afc79.opcodes_sh/src/proto_alpha/lib_protocol/script_interpreter.ml.html#L415"><code>GET</code></a>; the instructions <a href="https://research-development.nomadic-labs.com/files/test_coverage/_coverage.20191003.1717.0478afc79.opcodes_sh/src/proto_alpha/lib_protocol/script_interpreter.ml.html#L931"><code>SOURCE</code></a> and <a href="https://research-development.nomadic-labs.com/files/test_coverage/_coverage.20191003.1717.0478afc79.opcodes_sh/src/proto_alpha/lib_protocol/script_interpreter.ml.html#L934"><code>SENDER</code></a> related to the transaction in which a contract is executed; and arithmetic operations such as <code>INT</code> that converts an integer (<code>int</code>) to a natural number (<code>nat</code>).</p>
<p>The <a href="https://research-development.nomadic-labs.com/files/test_coverage/_coverage.20191003.1735.0478afc79.contracts_sh/src/proto_alpha/lib_protocol/script_interpreter.ml.html">coverage obtained when using the full Tezos node test suite</a>
is better, at 70.21% test coverage. However, it is still
worrying to see that some instructions, such as <a href="https://research-development.nomadic-labs.com/files/test_coverage/_coverage.20191003.1735.0478afc79.contracts_sh/src/proto_alpha/lib_protocol/script_interpreter.ml.html#L326"><code>SIZE</code> for lists</a> and
<a href="https://research-development.nomadic-labs.com/files/test_coverage/_coverage.20191003.1735.0478afc79.contracts_sh/src/proto_alpha/lib_protocol/script_interpreter.ml.html#L373"><code>MAP</code> for maps</a>, have no tests at all!</p>
<!-- For reference, coverage data for the full node is available here. -->
<h4 id="adding-tests"><a class="toclink" href="#adding-tests">Adding Tests</a></h4>
<p>There is a simple solution to the lacking coverage: writing more
tests. Fortunately, this is often quite straightforward using the
<a href="http://tezos.gitlab.io/developer/python_testing_framework.html">Python Execution and Testing Framework</a> for
Tezos. This framework provides a programmable <span class="caps">API</span> for the <code>tezos-client</code> and <code>tezos-node</code> binaries.
Writing tests is especially simple for instructions such as <a href="https://michelson.nomadic-labs.com/#instr-INT"><code>INT</code></a>,
that depend only on the state of the stack.
We will have to do some more work for instructions such as <a href="https://michelson.nomadic-labs.com/#instr-SOURCE"><code>SOURCE</code></a> that
interact with the block chain.</p>
<h5 id="testing-simple-instructions"><a class="toclink" href="#testing-simple-instructions">Testing simple instructions</a></h5>
<p>For simple instructions, our testing approach consists of
writing a simple contract that takes a sample input of the instruction by
parameter, and leaves the result of the instruction in storage. For
instance, to test the <code>INT</code> operation, we write the contract <code>int.tz</code>:</p>
<div class="highlight"><pre><span></span><code><span class="k">parameter</span><span class="w"> </span><span class="kt">nat</span><span class="p">;</span>
<span class="k">storage</span><span class="w"> </span><span class="p">(</span><span class="kt">option</span><span class="w"> </span><span class="kt">int</span><span class="p">);</span>
<span class="c1"># this contract takes a natural number as parameter, converts it to an</span>
<span class="c1"># integer and stores it.</span>
<span class="k">code</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">CAR</span><span class="p">;</span><span class="w"> </span><span class="kr">INT</span><span class="p">;</span><span class="w"> </span><span class="kr">SOME</span><span class="p">;</span><span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="kt">operation</span><span class="p">;</span><span class="w"> </span><span class="kr">PAIR</span><span class="w"> </span><span class="p">};</span>
</code></pre></div>
<p>Then, we write a pytest that executes the contract
with a sample input, retrieves the
resulting storage and asserts that it is equal to the expected one.
As we have a large number of tests like this, we use
<a href="https://docs.pytest.org/en/latest/parametrize.html"><code>pytest</code> parameterization</a>.
This enables the execution of the same test with different parameters.
In our case,
we write one test method that is parameterized by a
tuple <code>(contract, param, storage, expected)</code>.</p>
<div class="highlight"><pre><span></span><code> <span class="k">def</span> <span class="nf">test_contract_input_output</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">client</span><span class="p">,</span> <span class="n">contract</span><span class="p">,</span> <span class="n">param</span><span class="p">,</span> <span class="n">storage</span><span class="p">,</span> <span class="n">expected</span><span class="p">):</span>
<span class="n">contract</span> <span class="o">=</span> <span class="sa">f</span><span class="s1">'</span><span class="si">{</span><span class="n">CONTRACT_PATH</span><span class="si">}</span><span class="s1">/</span><span class="si">{</span><span class="n">contract</span><span class="si">}</span><span class="s1">'</span>
<span class="n">run_script_res</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">run_script</span><span class="p">(</span><span class="n">contract</span><span class="p">,</span> <span class="n">param</span><span class="p">,</span> <span class="n">storage</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">run_script_res</span><span class="o">.</span><span class="n">storage</span> <span class="o">==</span> <span class="n">expected</span>
</code></pre></div>
<p>It runs <code>contract</code>, using the <code>run script</code> command of
<code>tezos-client</code>. Here, <code>client</code> is a fixture provided by the Tezos
Python Execution and Testing framework, that wraps a <code>tezos-client</code>
binary. The <code>run script</code> command is called with the parameter <code>param</code>
and the initial storage <code>storage</code>. Then, the test verifies that after
execution, the final storage contains <code>expected</code>.</p>
<p>For instance, to test the <code>INT</code> instruction, we could instantiate the
test parameters with <code>('int.tz', 'None', '0', '(Some 0)')</code>, meaning that
the when the <code>int.tz</code> contract is passed <code>0</code> by parameter, then the
final storage should contain <code>(Some 0)</code>.
These instantiations are given by decorating the test method.
The resulting code looks like this:</p>
<div class="highlight"><pre><span></span><code> <span class="nd">@pytest</span><span class="o">.</span><span class="n">mark</span><span class="o">.</span><span class="n">parameterize</span><span class="p">(</span><span class="o">...</span><span class="p">,</span> <span class="p">[</span>
<span class="c1"># ...</span>
<span class="c1"># Test INT</span>
<span class="p">(</span><span class="s1">'int.tz'</span><span class="p">,</span> <span class="s1">'None'</span><span class="p">,</span> <span class="s1">'0'</span><span class="p">,</span> <span class="s1">'(Some 0)'</span><span class="p">),</span>
<span class="p">(</span><span class="s1">'int.tz'</span><span class="p">,</span> <span class="s1">'None'</span><span class="p">,</span> <span class="s1">'1'</span><span class="p">,</span> <span class="s1">'(Some 1)'</span><span class="p">),</span>
<span class="p">(</span><span class="s1">'int.tz'</span><span class="p">,</span> <span class="s1">'None'</span><span class="p">,</span> <span class="s1">'9999'</span><span class="p">,</span> <span class="s1">'(Some 9999)'</span><span class="p">),</span>
<span class="c1"># ...</span>
<span class="p">])</span>
<span class="k">def</span> <span class="nf">test_contract_input_output</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">client</span><span class="p">,</span> <span class="n">contract</span><span class="p">,</span> <span class="n">param</span><span class="p">,</span> <span class="n">storage</span><span class="p">,</span> <span class="n">expected</span><span class="p">):</span>
<span class="c1"># ...</span>
</code></pre></div>
<h5 id="more-involved-testing"><a class="toclink" href="#more-involved-testing">More involved testing</a></h5>
<p>Testing instructions that depend on the state of the block chain
requires more effort, as exemplified by our tests for the <a href="https://michelson.nomadic-labs.com/#instr-SOURCE"><code>SOURCE</code></a> instruction.
This instruction pushes the address of the contract that <em>initiated</em> the
current transaction, called the <strong>source</strong>. Note that this may differ from the <strong>sender</strong>,
which is the address of
the contract that initiated the current <em>internal</em> transaction and
that is obtained by the <code>SENDER</code> instruction.</p>
<p>Let us demonstrate the difference between the two instructions with a
scenario involving two contracts. The first
contract, <code>receiver</code>, puts the result of <code>SOURCE</code> in storage:</p>
<div class="highlight"><pre><span></span><code><span class="k">parameter</span><span class="w"> </span><span class="kt">unit</span><span class="p">;</span>
<span class="k">storage</span><span class="w"> </span><span class="kt">address</span><span class="p">;</span>
<span class="k">code</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">DROP</span><span class="p">;</span><span class="w"> </span><span class="kr">SOURCE</span><span class="p">;</span><span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="kt">operation</span><span class="p">;</span><span class="w"> </span><span class="kr">PAIR</span><span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>The second contract, <code>proxy</code>, transfers whatever amount
it receives to another contract, as specified by the parameter
of the transaction:</p>
<div class="highlight"><pre><span></span><code><span class="k">parameter</span><span class="w"> </span><span class="p">(</span><span class="kt">contract</span><span class="w"> </span><span class="kt">unit</span><span class="p">);</span>
<span class="k">storage</span><span class="w"> </span><span class="kt">unit</span><span class="p">;</span>
<span class="k">code</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="kr">UNPAIR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">AMOUNT</span><span class="p">;</span>
<span class="w"> </span><span class="kr">UNIT</span><span class="p">;</span>
<span class="w"> </span><span class="kr">TRANSFER_TOKENS</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="kt">operation</span><span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="kr">CONS</span><span class="p">;</span>
<span class="w"> </span><span class="kr">PAIR</span>
<span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>Now, assume that the contract <code>bootstrap2</code> creates a transaction to the <code>proxy</code> contract
sending 99 ꜩ and giving the address of the <code>receiver</code> contract as parameter . Then
the smart contract <code>proxy</code> creates an internal transaction that transfers the received 99 ꜩ to
<code>receiver</code>. This is illustrated by the following figure:</p>
<p><img alt="The proxy contract transfers money received from bootstrap2 to the receiver contract" src="https://research-development.nomadic-labs.com/images/test_coverage/proxy-receiver.png"></p>
<p>In this scenario, the <strong>source</strong> of the internal transaction (from
<code>proxy</code> to <code>receiver</code>) is <code>bootstrap2</code>, whereas the <strong>sender</strong> of that
transaction is the <code>proxy</code>.</p>
<p>We test <code>SOURCE</code> to verify that it operates as expected.
First, we test a direct transfer from <code>bootstrap2</code> to the <code>receiver</code>
contract, and then a transfer through the <code>proxy</code>
contract as in scenario described above. In both cases, we assert that
source address left in storage of <code>receiver</code> is that of <code>bootstrap2</code>:</p>
<div class="highlight"><pre><span></span><code> <span class="k">def</span> <span class="nf">test_source</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">client</span><span class="p">):</span>
<span class="n">init_store</span> <span class="o">=</span> <span class="n">IDENTITIES</span><span class="p">[</span><span class="s1">'bootstrap4'</span><span class="p">][</span><span class="s1">'identity'</span><span class="p">]</span>
<span class="n">init_with_transfer</span><span class="p">(</span><span class="n">client</span><span class="p">,</span>
<span class="sa">f</span><span class="s1">'</span><span class="si">{</span><span class="n">CONTRACT_PATH</span><span class="si">}</span><span class="s1">/receiver.tz'</span><span class="p">,</span>
<span class="sa">f</span><span class="s1">'"</span><span class="si">{</span><span class="n">init_store</span><span class="si">}</span><span class="s1">"'</span><span class="p">,</span>
<span class="mi">1000</span><span class="p">,</span> <span class="s1">'bootstrap1'</span><span class="p">)</span>
<span class="c1"># direct transfer to the contract</span>
<span class="n">client</span><span class="o">.</span><span class="n">transfer</span><span class="p">(</span><span class="mi">99</span><span class="p">,</span> <span class="s1">'bootstrap2'</span><span class="p">,</span> <span class="s1">'receiver'</span><span class="p">,</span> <span class="p">[</span><span class="s1">'--burn-cap'</span><span class="p">,</span> <span class="s1">'10'</span><span class="p">])</span>
<span class="n">bake</span><span class="p">(</span><span class="n">client</span><span class="p">)</span>
<span class="n">source_addr</span> <span class="o">=</span> <span class="n">IDENTITIES</span><span class="p">[</span><span class="s1">'bootstrap2'</span><span class="p">][</span><span class="s1">'identity'</span><span class="p">]</span>
<span class="n">assert_storage_contains</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="s1">'receiver'</span><span class="p">,</span> <span class="sa">f</span><span class="s1">'"</span><span class="si">{</span><span class="n">source_addr</span><span class="si">}</span><span class="s1">"'</span><span class="p">)</span>
<span class="c1"># indirect transfer to the contract through proxy</span>
<span class="n">contract_addr</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">get_contract_address</span><span class="p">(</span><span class="s1">'receiver'</span><span class="p">)</span>
<span class="n">client</span><span class="o">.</span><span class="n">transfer</span><span class="p">(</span><span class="mi">99</span><span class="p">,</span> <span class="s1">'bootstrap2'</span><span class="p">,</span> <span class="s1">'proxy'</span><span class="p">,</span>
<span class="p">[</span><span class="s1">'--burn-cap'</span><span class="p">,</span> <span class="s1">'10'</span><span class="p">,</span> <span class="s1">'--arg'</span><span class="p">,</span> <span class="sa">f</span><span class="s1">'"</span><span class="si">{</span><span class="n">contract_addr</span><span class="si">}</span><span class="s1">"'</span><span class="p">])</span>
<span class="n">bake</span><span class="p">(</span><span class="n">client</span><span class="p">)</span>
<span class="n">assert_storage_contains</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="s1">'receiver'</span><span class="p">,</span> <span class="sa">f</span><span class="s1">'"</span><span class="si">{</span><span class="n">source_addr</span><span class="si">}</span><span class="s1">"'</span><span class="p">)</span>
</code></pre></div>
<h4 id="final-results"><a class="toclink" href="#final-results">Final results</a></h4>
<p>From the beginning, we’ve had a reasonable degree of testing in place;
which was reflected in the core’s stability. Despite this, we committed
to achieving 100% (or close to it) of code coverage. To that end, since
April 2019, we have added 719<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup> new Python integration tests, using 66 new test
contracts. As a result of this effort, we now obtain a
<a href="https://research-development.nomadic-labs.com/files/test_coverage/_coverage.20200708.1016.ec198d588b6c438246dac9dda30449d20510891f.contracts/src/proto_alpha/lib_protocol/script_interpreter.ml.html">code coverage of 99.12%</a><sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup>
for the tests for the instruction integration tests and <a href="https://research-development.nomadic-labs.com/files/test_coverage/_coverage.20200708.1035.ec198d588b6c438246dac9dda30449d20510891f.full/src/proto_alpha/lib_protocol/script_interpreter.ml.html">100% for the full contract test
suite</a>. We cover all branches of the interpreter function in
<code>script_interpreter.ml</code>.</p>
<p>In addition to quantifying the confidence in the test suite of the
interpreter, <code>bisect_ppx</code> helped us discover dead code in the
interpreter. The interpreter contained a “peephole optimization” for
the <code>UNPAIR</code> macro.
This optimization was supposed to match the expansion of the <code>UNPAIR</code>
macro and transform directly a stack on the form <code>(Pair a b) : S</code> to
a stack of form <code>a : b : S</code>, without executing the full expansion.
To our surprise, we discovered that this
branch of the interpreter is never visited during testing. Quite
unexpectedly, since the <code>UNPAIR</code> macro is very commonly used in our test contracts.</p>
<p>It turns out that the optimization of <code>UNPAIR</code> was never applied. The
expansion of this macro had been changed since the optimization was
added to the interpreter, but the optimization had not been updated to
match. Consequently, the pattern match in interpreter no longer
matched the actual definition of <code>UNPAIR</code>. Thanks to <code>bisect_ppx</code>, we
were able to detect this dead code. As <a href="https://gitlab.com/nomadic-labs/tezos/-/merge_requests/73">gas updates in
Babylon</a>
removes much of the gains from this optimization, we <a href="http://tezos.gitlab.io/protocols/006_carthage.html#dead-optimisation-of-the-unpair-macro">removed it in
the current Carthage
protocol</a>.</p>
<h3 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h3>
<p>Now that we have achieved a high level of code coverage in the
interpreter, the next goal is to maintain this level and to extend it
to remaining parts of the Tezos code base.</p>
<p>For the first objective, we have added code coverage measurements to
our <a href="https://gitlab.com/tezos/tezos/-/merge_requests/1404">continuous integration
system</a>. This
lets developers calculate test coverage through a button push in the
<span class="caps">CI</span>’s interface. In the future, we would also like a developer to be
automatically informed if, for example, they add a new Michelson instruction but
forget to add tests verifying its implementation.</p>
<p>For the second objective, we are working on covering fully the type
checker of Michelson, <a href="https://research-development.nomadic-labs.com/files/test_coverage/_coverage.20200708.1035.ec198d588b6c438246dac9dda30449d20510891f.full/src/proto_alpha/lib_protocol/script_ir_translator.ml.html">currently at 75.94%
percent</a>. To
achieve this goal, we are writing both <a href="https://gitlab.com/tezos/tezos/-/merge_requests/980">unit
tests</a> and
<a href="https://gitlab.com/tezos/tezos/-/merge_requests/1939">integration
tests</a>.
We <a href="https://gitlab.com/tezos/tezos/-/merge_requests/1957">are also encouraging
developers</a> to
add automatic tests for all new functionality and when modifying existing behaviors.</p>
<h3 id="end-notes"><a class="toclink" href="#end-notes">End Notes</a></h3>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>For Python test cases, this counts the difference between test
items collected by pytest when running <code>pytest
tests_python/tests/test_contract*</code> in commit <code>#0478afc7</code> and in commit
<code>#395bee9dc</code>. For test contracts, this counts the difference between the
number of <code>*.tz</code> files present in the <code>src/bin_client/test/</code> respectively <code>tests_python/contracts</code> in
commit <code>#0478afc7</code> respectively commit <code>#395bee9dc</code>. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>The missing <a href="https://research-development.nomadic-labs.com/files/test_coverage/_coverage.20200708.1016.ec198d588b6c438246dac9dda30449d20510891f.contracts/src/proto_alpha/lib_protocol/script_interpreter.ml.html#L1346">0.88%
percent</a>
comes from a case of the deprecated <code>STEPS_TO_QUOTA</code> instruction which
is unreachable through the <code>tezos-client</code> commands used for
integration testing of contracts. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>Editor’s Note on Emmy+ Analysis2020-06-26T15:00:00+02:002020-06-26T15:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2020-06-26:/editors-note-on-emmy-analysis.html<p>Prompted by a discussion initiated by Michael Neuder from University of Colorado Boulder, we have revised the estimated number of confirmations in a particular scenario from <a href="https://research-development.nomadic-labs.com/analysis-of-emmy.html">the analysis of Emmy+</a>.</p>
<p>Please refer to the first paragraph for details.</p>Meanwhile at Nomadic Labs #82020-06-08T15:00:00+02:002020-06-08T15:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2020-06-08:/meanwhile-at-nomadic-labs-8.html<p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>Let’s talk about what’s been going on at Nomadic Labs for the past two months. Despite the global pandemic, we’ve stayed hard at work for the Tezos Community.</p>
<p>We published an in-depth look on how we use regression testing to catch bugs in Tezos: <a href="https://research-development.nomadic-labs.com/catching-sneaky-regressions-with-pytest-regtest.html">Catching sneaky regressions with pytest-regtest</a>.</p>
<p>Marco Stronati gave a speech about the integration of Sapling in Tezos for privacy and compliance at the <a href="https://readylayer.one/">Ready Layer One conference</a>.</p>
<p>May was a big month as we introduced the new Multi-network node that we’ve been working on for a while. More information on the critical update including the new multi network node can be found <a href="https://forum.tezosagora.org/t/critical-reliability-update-for-all-tezos-nodes/1868">here</a>.</p>
<p>We also released a critical reliability update for all Tezos nodes. This update fixed a bug that could cause nodes to crash with a segmentation fault. A newsletter was sent out to all subscribers that recommends everybody to update, which included instructions. Please subscribe to our newsletter <a href="https://lists.nomadic-labs.com/subscription/AnDEIwStd">here</a>, if you haven’t already.</p>
<p>We participated in a webinar that introduced Tezos hosted by the <a href="https://bsaepfl.ch/">blockchain student association</a>. </p>
<p>Due to the many questions related to ongoing protocol development, we hosted <span class="amp">&</span> participated in a number of AMAs. The first of which was held on May 12th on Reddit. The questions <span class="amp">&</span> answers can be viewed <a href="https://www.reddit.com/r/tezos/comments/ghrjv3/nomadic_labs_ama_on_rtezos_may_12th_1pm_est/">here</a>.
A second <span class="caps">AMA</span> was held on Agora on May 26th together with Metastate. Questions and answers can be found <a href="https://forum.tezosagora.org/t/tezos-core-development-ama-2-nomadic-labs-metastate-007/1923">here</a>.</p>
<p>Together with other actors of the Tezos ecosystem, Marco Stronati and Marc Beunardeau spoke on the <a href="https://www.zeroknowledge.fm/130">Zeroknowledge podcast</a>.
The discussion included the Tezos community, governance, and how Zcash’s Zero Knowledge Proofs (Sapling) will be integrated in an upcoming Tezos protocol upgrade proposal.</p>
<p>Apart from the above we have been busy with work on the shell, core protocol and mainly the upcoming protocol upgrade which will be ready for injection sometime soon.</p>
<p>Thanks all! More to come.</p>Meanwhile at Nomadic Labs #72020-04-09T17:00:00+02:002020-04-09T17:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2020-04-09:/meanwhile-at-nomadic-labs-7.html<p>Summary:</p>
<p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>It’s time to talk about what has been going on at Nomadic Labs for the past two
months. Many of you might know that a lot of us at Nomadic Labs already work
remotely on a regular basis. That being said, the situation with the
coronavirus hasn’t …</p><p>Summary:</p>
<p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>It’s time to talk about what has been going on at Nomadic Labs for the past two
months. Many of you might know that a lot of us at Nomadic Labs already work
remotely on a regular basis. That being said, the situation with the
coronavirus hasn’t had a chance to disrupt our work. Some Conferences were
rescheduled, others are held remotely and our work at Nomadic Labs continues as usual. </p>
<p>The month of February was packed with four Tezos training sessions organized by
Nomadic Labs. Topics included a general introduction to the structure of the
project, consensus, fees and gas in Michelson, zero knowledge proofs, how to
deploy a node and much more. The entire program can be viewed
<a href="https://training.nomadic-labs.com/training.html">here</a>. You can follow the
individual links to view <span class="caps">PDF</span>’s of the presentations that were held.</p>
<p>On the 13th of February we participated in an <span class="caps">AMA</span> on protocol development
featuring Nomadic Labs, Cryptium Labs, and Dailambda. The thread with all
questions and answers is available
<a href="https://forum.tezosagora.org/t/tezos-core-development-ama-nomadic-labs-cryptium-labs-dailambda/1604/93">here</a>.</p>
<p>Philippe Bidinger and Raphaël Cauderlier attended the
<a href="https://fc20.ifca.ai/">financial cryptography and security conference 2020</a>
on February 14th. Raphaël presented
<a href="https://arxiv.org/pdf/2001.02630.pdf">“Albert, an intermediate smart contract language for Tezos”</a>.</p>
<p>In mid-February, we announced a new version of the mainnet-staging branch and
the discontinuation of the mainnet-lmdb branch. We recommend that all bakers
still running the mainnet-lmdb branch upgrade to the mainnet-staging branch as
soon as possible. The announcement was sent out in our newsletter and can also
be viewed
<a href="https://forum.tezosagora.org/t/attention-to-all-bakers-a-new-version-of-the-mainnet-staging-branch-was-released/1649">here</a>.
We would like to remind everybody that all changes also always get communicated
on the <a href="https://tezos-baking.slack.com">Obsidian’s baking slack channel</a>, on
<a href="https://twitter.com/LabosNomades">our twitter account</a> and on the
<a href="https://www.reddit.com/r/tezos/">Tezos subreddit</a>.</p>
<p>On February 26th, Pietro Abate attended the
<a href="https://www.meetup.com/Tezos-Barcelona-Meetups/events/268460321/">Barcelona Tezos meetup</a>
sponsored by Nomadic Labs and Tezos Commons. Attendees learned more details
about Tenderbake and how to run a Tezos node.</p>
<p>Carthage successfully activated and became the new Tezos protocol on March 5th.
The upgrade process was smooth with no issues.</p>
<p>We were also pleased to take part in the <span class="caps">ETHCC</span> conference in the beginning of
March, although some of us did not make it due to current travel restrictions.</p>
<p>Shortly after <span class="caps">ETHCC</span> on March 6th, we organized a Tezos Developer Day. The
entire event can be viewed on our
<a href="https://www.youtube.com/watch?v=Y_qg69d9owo">YouTube channel</a>.</p>
<p>For individual segments please see the links below:</p>
<ul>
<li><a href="https://www.youtube.com/watch?v=JqGPY_f4kWs">Alexandre Doussot - Nomadic Labs</a></li>
<li><a href="https://www.youtube.com/watch?v=HjjSl4oSCNU">Adrian Brink – Tezos vs Ethereum</a></li>
<li><a href="https://www.youtube.com/watch?v=I34vCa9kCGY">Christian Rinderknecht + Rémi Lesénéchal – <span class="caps">LIGO</span></a></li>
<li><a href="https://www.youtube.com/watch?v=Ez3SQU_C2fY">Laurent Kratz – NeoFacto</a></li>
<li><a href="https://www.youtube.com/watch?v=NIFMICMXuhc">Aymeric Bettencourt - Octo Technology</a></li>
<li><a href="https://www.youtube.com/watch?v=7_gWun-lzOk">Laszlo Szabo – Skillz</a></li>
<li><a href="https://www.youtube.com/watch?v=mDUEA_rFpyU">Alain Broustail - Sword Group</a></li>
<li><a href="https://www.youtube.com/watch?v=9Y_ZhsBu9RU">Frederic Bertoïa – Equisafe</a></li>
<li><a href="https://www.youtube.com/watch?v=6ZMtjV9m7sw">Roland Zumkeller – SmartPy</a></li>
<li><a href="https://www.youtube.com/watch?v=qJFqXpXwKgI">Julian Konchunas, Anastasiia Kondaurov, Sergii Glushkovskyi - Madfish Solution</a></li>
<li><a href="https://www.youtube.com/watch?v=M0--mzEBwe8">Matej Sima - Stove Labs</a></li>
<li><a href="https://www.youtube.com/watch?v=1MWn9v7VrUU">James Haver - CamlCase</a></li>
<li><a href="https://www.youtube.com/watch?v=_2LN5TlQdOc">Cruz Molina - Truffle and Tezos</a></li>
</ul>
<p>On March 26th, we organized a “Sapling integration in Tezos” <span class="caps">AMA</span> on the Agora
Forum. All question and answers can be viewed in this
<a href="https://forum.tezosagora.org/t/sapling-integration-in-tezos-ama-nomadic-labs/1775">thread</a></p>
<p>Marco Stronati attended the
<a href="https://virtual.tezos.berlin/">Virtual Tezos Berlin Meetup</a>
on March 28th. These two videos feature Marco: </p>
<ul>
<li><a href="https://www.youtube.com/watch?v=67WppWUlPf4">Tezos Sapling Integration</a></li>
<li><a href="https://www.youtube.com/watch?v=IYfa1kT847M">Tezos Protocol 007 Fireside Chat</a></li>
</ul>
<p>Stay tuned for more updates next month soon.</p>Catching sneaky regressions with pytest-regtest2020-03-24T15:00:00+01:002020-03-24T15:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2020-03-24:/catching-sneaky-regressions-with-pytest-regtest.html<p>An in-depth look on how we use regression testing to catch bugs in Tezos.</p><p>Testing is an important complement to formal methods that we use
through out Tezos to ensure software quality. In this blog post we
discuss how we use <em>regression testing</em>, through the
<a href="https://pypi.org/project/pytest-regtest/"><code>pytest-regtest</code></a> tool.</p>
<h2 id="regression-testing"><a class="toclink" href="#regression-testing">Regression testing</a></h2>
<p>Regression testing is a coarse-grained testing method for detecting
unintended changes in the behavior of the system under test. We have
applied regression testing to the <code>tezos-client</code>, notably to
smart-contract type checking and execution.
Globally, regression testing consists of two components. First, a way
of running and storing the behavior of a system in a test
scenario. Second, a way of running the same test again and comparing
the behavior of the first and second run.</p>
<p>This is why we call regression testing <em>coarse-grained</em>: a regression
test passes only if the output is <em>exactly</em> the same as in the first,
stored, run. Consequently, a regression test will detect
non-functional changes, such as changes in white space output by the
tested system. Such nitpicking can be overly restrictive, and we will
show later in this blog post how to relax overly strict verification.</p>
<p>In our case, we have enabled regression testing on an existing test
suite that exercises the type checker and interpreter of Michelson
smart contracts. In the first run of the test suite, the output of
each command passed to <code>tezos-client</code> by the test case is stored in a
log file by the test runner. After verifying that validity of the
output, we commit these log files to source control. In subsequent
runs, the output of each command sent to the client is compared to the
expected output stored in the corresponding log file.</p>
<h3 id="regression-testing-using-pytest-regtest"><a class="toclink" href="#regression-testing-using-pytest-regtest">Regression testing using <code>pytest-regtest</code></a></h3>
<p>To implement regression tests
we plug in
<a href="https://pypi.org/project/pytest-regtest/"><code>pytest-regtest</code></a>
to the <code>pytest</code>-based <a href="http://tezos.gitlab.io/developer/python_testing_framework.html">Python Testing and Execution framework</a> used for integration testing of Tezos.
This plugin provides the <code>regtest</code> fixture that is then used as a file object: it can be written to, or used as a <a href="https://docs.python.org/3/library/stdtypes.html#typecontextmanager">context manager</a>.
If the <code>--regtest-reset</code> flag is passed to pytest, then all output
written to the <code>regtest</code> fixture during a given test case is stored to
a log file. On subsequent runs (unless the <code>--regtest-reset</code> flag is
passed again), the output written to the fixture will be compared to
what was recorded during the first run.</p>
<p>Let’s write a test case and then extend it with regression testing.
We start out with a classic unit test that exercises the Michelson
<code>ADD</code> instruction. We first define a Michelson contract that takes a
pair of natural numbers as parameter and, using the <code>ADD</code> instruction,
calculates their sum and puts it in storage:</p>
<div class="highlight"><pre><span></span><code><span class="k">parameter</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="kt">nat</span><span class="w"> </span><span class="kt">nat</span><span class="p">);</span>
<span class="k">storage</span><span class="w"> </span><span class="p">(</span><span class="kt">option</span><span class="w"> </span><span class="kt">nat</span><span class="p">);</span>
<span class="k">code</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">CAR</span><span class="p">;</span><span class="w"> </span><span class="kr">UNPAIR</span><span class="p">;</span><span class="w"> </span><span class="kr">ADD</span><span class="p">;</span><span class="w"> </span><span class="kr">SOME</span><span class="p">;</span><span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="kt">operation</span><span class="p">;</span><span class="w"> </span><span class="kr">PAIR</span><span class="p">;</span><span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>We store the contract in <code>add_example.tz</code> and write the following pytest:</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="nc">TestRegressionExample</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">test_example_add</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">client</span><span class="p">):</span>
<span class="n">add_contract</span> <span class="o">=</span> <span class="s1">'add_example.tz'</span>
<span class="n">run_script_res</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">run_script</span><span class="p">(</span>
<span class="n">add_contract</span><span class="p">,</span> <span class="s1">'None'</span><span class="p">,</span> <span class="s1">'Pair 10 15'</span><span class="p">,</span> <span class="n">amount</span><span class="o">=</span><span class="kc">None</span>
<span class="p">)</span>
<span class="k">assert</span> <span class="n">run_script_res</span><span class="o">.</span><span class="n">storage</span> <span class="o">==</span> <span class="s1">'(Some 25)'</span>
</code></pre></div>
<p>The tests runs the <code>add_example.tz</code> contract with
parameters <code>10</code> and <code>15</code> using the
<code>run_script</code> method of the <code>client</code> fixture of the Tezos Python
testing suite, and then verifies that the result is indeed <code>25</code>.</p>
<p>As expected, this test passes:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>-rf<span class="w"> </span>tests_python/tests/test_contract_opcodes.py<span class="w"> </span>-k<span class="w"> </span><span class="nv">test_example_add</span>
<span class="o">==============================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.7.3,<span class="w"> </span>pytest-4.4.0,<span class="w"> </span>py-1.8.0,<span class="w"> </span>pluggy-0.9.0<span class="w"> </span>--<span class="w"> </span>/usr/bin/python3
cachedir:<span class="w"> </span>.pytest_cache
rootdir:<span class="w"> </span>.../tezos/tests_python,<span class="w"> </span>inifile:<span class="w"> </span>pytest.ini
plugins:<span class="w"> </span>timeout-1.3.3,<span class="w"> </span>parallel-0.0.9
collected<span class="w"> </span><span class="m">302</span><span class="w"> </span>items<span class="w"> </span>/<span class="w"> </span><span class="m">301</span><span class="w"> </span>deselected<span class="w"> </span>/<span class="w"> </span><span class="m">1</span><span class="w"> </span>selected
tests_python/tests/test_contract_opcodes.py::TestContractOpcodes::test_example_add<span class="w"> </span>PASSED
</code></pre></div>
<p>This is all very well, but say that we want to test some other aspects
of the <code>ADD</code> instruction, such as its gas consumption. We can either
write a new test case, but this quickly gets tiresome as Michelson has
some 80 instructions that should all be tested. Furthermore, gas
constants are likely to evolve, so hard coding them in the tests makes
for brittle tests. Our test suite already contains tests for all
instructions — it would be more convenient if we could piggy-back
existing tests, but without asserting explicitly the gas costs of
each instruction in those tests.</p>
<p>First, how do we inspect the gas consumption of a contract? We can track
gas consumption of a program (and by extension, its instructions)
using the <code>--trace-stack</code> option to the <code>run script</code> command of the
client. When this option is given, the interpreter reports a stack
trace that contains the contents of the stack and the remaining gas at
each program point during execution. Let’s try (the output has
been redacted for brevity):</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-client<span class="w"> </span>run<span class="w"> </span>script<span class="w"> </span>add_example.tz<span class="w"> </span>on<span class="w"> </span>storage<span class="w"> </span>None<span class="w"> </span>and<span class="w"> </span>input<span class="w"> </span><span class="s1">'Pair 10 15'</span><span class="w"> </span>--trace-stack
storage
<span class="w"> </span><span class="o">(</span>Some<span class="w"> </span><span class="m">25</span><span class="o">)</span>
emitted<span class="w"> </span>operations
big_map<span class="w"> </span>diff
trace
<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">9</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799557</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="o">(</span>Pair<span class="w"> </span><span class="o">(</span>Pair<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">15</span><span class="o">)</span><span class="w"> </span>None<span class="o">)</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799557</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="o">(</span>Pair<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">15</span><span class="o">)</span><span class="w"> </span>@parameter<span class="w"> </span><span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">13</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799556</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="o">(</span>Pair<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">15</span><span class="o">)</span><span class="w"> </span>@parameter
<span class="w"> </span><span class="o">(</span>Pair<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">15</span><span class="o">)</span><span class="w"> </span>@parameter<span class="w"> </span><span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">14</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799556</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
<span class="o">[</span>...<span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">11</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799554</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">10</span>
<span class="w"> </span><span class="m">15</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">18</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799553</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">25</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">19</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799553</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="o">(</span>Some<span class="w"> </span><span class="m">25</span><span class="o">)</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">20</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799552</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="o">{}</span>
<span class="w"> </span><span class="o">(</span>Some<span class="w"> </span><span class="m">25</span><span class="o">)</span><span class="w"> </span><span class="o">]</span>
<span class="o">[</span>...<span class="o">]</span>
</code></pre></div>
<p>We can deduce that the gas consumption for the <code>ADD</code> instruction in this
execution is less than one gas unit (location 18), and that the gas consumption for the
whole script is no more than five units. This is correct, and we
would like future tests to signal if this changes unexpectedly. A
solution would be to write the whole stack trace to the <code>regtest</code> fixture.
To implement this idea, we
modify our test:</p>
<div class="highlight"><pre><span></span><code> <span class="k">def</span> <span class="nf">test_example_add_reg</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">client</span><span class="p">,</span> <span class="n">regtest</span><span class="p">):</span>
<span class="n">add_contract</span> <span class="o">=</span> <span class="s2">"add_example.tz"</span>
<span class="n">run_script_res</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">run_script</span><span class="p">(</span>
<span class="n">add_contract</span><span class="p">,</span> <span class="s1">'None'</span><span class="p">,</span> <span class="s1">'Pair 10 15'</span><span class="p">,</span> <span class="n">amount</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">trace_stack</span><span class="o">=</span><span class="kc">True</span>
<span class="p">)</span>
<span class="n">regtest</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">run_script_res</span><span class="o">.</span><span class="n">client_output</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">run_script_res</span><span class="o">.</span><span class="n">storage</span> <span class="o">==</span> <span class="s1">'(Some 25)'</span>
</code></pre></div>
<p>The modified test requires the <code>regtest</code> fixture by including it as a
formal parameter. Then we instruct the client fixture to pass the
<code>--trace-stack</code> flag to <code>tezos-client</code> through the <code>trace_stack=True</code>.
Finally, we write the complete output from the client, as obtained
from <code>run_script_res.client_output</code>, to <code>regtest</code>. Now if we run the
<code>test_example_add_reg</code>, it will fail due to the lack of a pre-existing
log file.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>tests_python/tests/test_regression_example.py<span class="w"> </span>-k<span class="w"> </span>example_add_reg
tests_python/tests/test_regression_example.py::TestRegressionExample::test_example_add_reg<span class="w"> </span>FAILED<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">==================</span><span class="w"> </span><span class="nv">FAILURES</span><span class="w"> </span><span class="o">======================</span>
____________________________<span class="w"> </span>TestRegressionExample.test_example_add_reg<span class="w"> </span>____________________________
regression<span class="w"> </span><span class="nb">test</span><span class="w"> </span>output<span class="w"> </span>differences<span class="w"> </span><span class="k">for</span><span class="w"> </span>tests/test_regression_example.py::TestRegressionExample::test_example_add_reg:
><span class="w"> </span>---<span class="w"> </span>current
><span class="w"> </span>+++<span class="w"> </span>tobe
><span class="w"> </span>@@<span class="w"> </span>-1,43<span class="w"> </span>+1<span class="w"> </span>@@
><span class="w"> </span>-storage
><span class="w"> </span>-<span class="w"> </span><span class="o">(</span>Some<span class="w"> </span><span class="m">25</span><span class="o">)</span>
><span class="w"> </span>-emitted<span class="w"> </span>operations
>
><span class="w"> </span>-big_map<span class="w"> </span>diff
<span class="o">[</span>...<span class="o">]</span>
</code></pre></div>
<p>We try running again with the <code>--regtest-reset</code> to create the log file:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>--regtest-reset<span class="w"> </span>tests_python/tests/test_regression_example.py<span class="w"> </span>-k<span class="w"> </span>example_add_reg
</code></pre></div>
<p>The generated log file is found in <code>tests_python/tests/_regtest_outputs/test_regression_example.TestRegressionExample\:\:test_example_add_reg.out</code>, and we can verify that it contains the expected stack trace:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>tests_python/tests/_regtest_outputs/test_regression_example.TestRegressionExample<span class="se">\:\:</span>test_example_add_reg.out
storage
<span class="w"> </span><span class="o">(</span>Some<span class="w"> </span><span class="m">25</span><span class="o">)</span>
emitted<span class="w"> </span>operations
big_map<span class="w"> </span>diff
trace
<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">9</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799557</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="o">(</span>Pair<span class="w"> </span><span class="o">(</span>Pair<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">15</span><span class="o">)</span><span class="w"> </span>None<span class="o">)</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799557</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="o">(</span>Pair<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">15</span><span class="o">)</span><span class="w"> </span>@parameter<span class="w"> </span><span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">13</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799556</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="o">(</span>Pair<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">15</span><span class="o">)</span><span class="w"> </span>@parameter
<span class="w"> </span><span class="o">(</span>Pair<span class="w"> </span><span class="m">10</span><span class="w"> </span><span class="m">15</span><span class="o">)</span><span class="w"> </span>@parameter<span class="w"> </span><span class="o">]</span>
<span class="o">[</span>...<span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">11</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799554</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">10</span>
<span class="w"> </span><span class="m">15</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">18</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799538</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">25</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">19</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799537</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
</code></pre></div>
<p>We can now run the regression test and verify that it passes:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>tests_python/tests/test_regression_example.py<span class="w"> </span>-k<span class="w"> </span><span class="nv">example_add_reg</span>
<span class="o">===================================================================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.7.3,<span class="w"> </span>pytest-4.4.0,<span class="w"> </span>py-1.8.0,<span class="w"> </span>pluggy-0.9.0<span class="w"> </span>--<span class="w"> </span>/usr/bin/python3
cachedir:<span class="w"> </span>.pytest_cache
rootdir:<span class="w"> </span>.../tezos/tests_python,<span class="w"> </span>inifile:<span class="w"> </span>pytest.ini
plugins:<span class="w"> </span>timeout-1.3.3,<span class="w"> </span>parallel-0.0.9,<span class="w"> </span>regtest-1.4.2
collected<span class="w"> </span><span class="m">2</span><span class="w"> </span>items<span class="w"> </span>/<span class="w"> </span><span class="m">1</span><span class="w"> </span>deselected<span class="w"> </span>/<span class="w"> </span><span class="m">1</span><span class="w"> </span>selected
tests_python/tests/test_regression_example.py::TestRegressionExample::test_example_add_reg<span class="w"> </span>PASSED<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">==========================</span><span class="w"> </span><span class="m">1</span><span class="w"> </span>passed,<span class="w"> </span><span class="m">1</span><span class="w"> </span>deselected<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="m">2</span>.29<span class="w"> </span><span class="nv">seconds</span><span class="w"> </span><span class="o">================================================</span>
</code></pre></div>
<p>Now if a developer inadvertently introduces a change that modifies the
gas consumption of the <code>ADD</code> instruction or its functioning, then
this regression test should break. Likewise if the developer intentionally
changes the semantics of the instruction. However, this should not be taken for
granted. A good practice when writing tests is <em>to test the test itself</em>
by intentionally breaking the functionality tested
and verifying that the corresponding test breaks. We follow
this advice and introduce a change in the gas cost of additions.</p>
<p>Gas costs are (mainly) defined in
<code>src/proto_alpha/lib_protocol/michelson_v1_gas.ml</code>. Assume a cat,
well-meaning but imprudent, prances over the developer’s
keyboard, introducing the following change:</p>
<div class="highlight"><pre><span></span><code>diff<span class="w"> </span>--git<span class="w"> </span>a/src/proto_alpha/lib_protocol/michelson_v1_gas.ml<span class="w"> </span>b/src/proto_alpha/lib_protocol/michelson_v1_gas.ml
index<span class="w"> </span>f61e519fe..228af62c6<span class="w"> </span><span class="m">100644</span>
---<span class="w"> </span>a/src/proto_alpha/lib_protocol/michelson_v1_gas.ml
+++<span class="w"> </span>b/src/proto_alpha/lib_protocol/michelson_v1_gas.ml
@@<span class="w"> </span>-182,7<span class="w"> </span>+182,7<span class="w"> </span>@@<span class="w"> </span>module<span class="w"> </span><span class="nv">Cost_of</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>struct
<span class="w"> </span><span class="nb">let</span><span class="w"> </span>abs<span class="w"> </span><span class="nv">int</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>atomic_step_cost<span class="w"> </span><span class="o">(</span><span class="m">61</span><span class="w"> </span>+<span class="w"> </span><span class="o">((</span>int_bytes<span class="w"> </span>int<span class="o">)</span><span class="w"> </span>/<span class="w"> </span><span class="m">70</span><span class="o">))</span>
<span class="w"> </span><span class="nb">let</span><span class="w"> </span>int<span class="w"> </span><span class="nv">_int</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>free
<span class="w"> </span><span class="nb">let</span><span class="w"> </span><span class="nv">neg</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>abs
-<span class="w"> </span><span class="nb">let</span><span class="w"> </span>add<span class="w"> </span>i1<span class="w"> </span><span class="nv">i2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>atomic_step_cost<span class="w"> </span><span class="o">(</span><span class="m">51</span><span class="w"> </span>+<span class="w"> </span><span class="o">(</span>Compare.Int.max<span class="w"> </span><span class="o">(</span>int_bytes<span class="w"> </span>i1<span class="o">)</span><span class="w"> </span><span class="o">(</span>int_bytes<span class="w"> </span>i2<span class="o">)</span><span class="w"> </span>/<span class="w"> </span><span class="m">62</span><span class="o">))</span>
+<span class="w"> </span><span class="nb">let</span><span class="w"> </span>add<span class="w"> </span>i1<span class="w"> </span><span class="nv">i2</span><span class="w"> </span><span class="o">=</span><span class="w"> </span>atomic_step_cost<span class="w"> </span><span class="o">(</span><span class="m">9000</span><span class="w"> </span>+<span class="w"> </span><span class="m">51</span><span class="w"> </span>+<span class="w"> </span><span class="o">(</span>Compare.Int.max<span class="w"> </span><span class="o">(</span>int_bytes<span class="w"> </span>i1<span class="o">)</span><span class="w"> </span><span class="o">(</span>int_bytes<span class="w"> </span>i2<span class="o">)</span><span class="w"> </span>/<span class="w"> </span><span class="m">62</span><span class="o">))</span>
</code></pre></div>
<p>Oops! Suddenly additions have a fixed overhead of over 9000!
It would be unfortunate if this change passes our test suite. Let us
verify that it is not the case. We recompile the node and run the
tests again:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>pytest<span class="w"> </span>tests_python/tests/test_regression_example.py<span class="w"> </span>-k<span class="w"> </span><span class="nv">example_add_reg</span>
<span class="o">===================================================================</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>session<span class="w"> </span><span class="nv">starts</span><span class="w"> </span><span class="o">===========================</span>
platform<span class="w"> </span>linux<span class="w"> </span>--<span class="w"> </span>Python<span class="w"> </span><span class="m">3</span>.7.3,<span class="w"> </span>pytest-4.4.0,<span class="w"> </span>py-1.8.0,<span class="w"> </span>pluggy-0.9.0<span class="w"> </span>--<span class="w"> </span>/usr/bin/python3
cachedir:<span class="w"> </span>.pytest_cache
rootdir:<span class="w"> </span>.../tezos/tests_python,<span class="w"> </span>inifile:<span class="w"> </span>pytest.ini
plugins:<span class="w"> </span>timeout-1.3.3,<span class="w"> </span>parallel-0.0.9,<span class="w"> </span>regtest-1.4.2
collected<span class="w"> </span><span class="m">2</span><span class="w"> </span>items<span class="w"> </span>/<span class="w"> </span><span class="m">1</span><span class="w"> </span>deselected<span class="w"> </span>/<span class="w"> </span><span class="m">1</span><span class="w"> </span>selected
tests_python/tests/test_regression_example.py::TestRegressionExample::test_example_add_reg<span class="w"> </span>FAILED<span class="w"> </span><span class="o">[</span><span class="m">100</span>%<span class="o">]</span>
<span class="o">==================</span><span class="w"> </span><span class="nv">FAILURES</span><span class="w"> </span><span class="o">======================</span>
____________________________<span class="w"> </span>TestRegressionExample.test_example_add_reg<span class="w"> </span>____________________________
regression<span class="w"> </span><span class="nb">test</span><span class="w"> </span>output<span class="w"> </span>differences<span class="w"> </span><span class="k">for</span><span class="w"> </span>tests/test_regression_example.py::TestRegressionExample::test_example_add_reg:
><span class="w"> </span>---<span class="w"> </span>current
><span class="w"> </span>+++<span class="w"> </span>tobe
><span class="w"> </span>@@<span class="w"> </span>-28,16<span class="w"> </span>+28,16<span class="w"> </span>@@
><span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">11</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799554</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
><span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="m">10</span>
><span class="w"> </span><span class="m">15</span><span class="w"> </span><span class="o">]</span>
><span class="w"> </span>-<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">18</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799413</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
><span class="w"> </span>+<span class="w"> </span>-<span class="w"> </span>location:<span class="w"> </span><span class="m">18</span><span class="w"> </span><span class="o">(</span>remaining<span class="w"> </span>gas:<span class="w"> </span><span class="m">799538</span><span class="w"> </span>units<span class="w"> </span>remaining<span class="o">)</span>
</code></pre></div>
<p>All is well: the change is reflected in the output and the tests do
indeed break. Curiously, the difference between the previous gas
consumption and the new is not 9000, but <code>799538 - 799413 = 125</code>. The
reason is that the <code>atomic_step_cost</code> works with an internal gas unit,
which is later scaled up to obtain the user-facing gas unit.</p>
<h3 id="an-eavesdropping-client"><a class="toclink" href="#an-eavesdropping-client">An eavesdropping client</a></h3>
<p>In order to avoid modifying all existing test-cases and to specify
manually the output that should be registered in each test, we wrote a
new pytest fixture <code>client_regtest</code>. This fixture mixes the existing
<code>client</code> fixture with the <code>regtest</code> fixture of <code>pytest-regtest</code>. In
addition, it automatically stores the output of all commands passed to
<code>tezos-client</code>. For example, we now express the example test case:</p>
<div class="highlight"><pre><span></span><code> <span class="k">def</span> <span class="nf">test_example_add_reg_client_regtest</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">client_regtest</span><span class="p">):</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">client_regtest</span>
<span class="n">add_contract</span> <span class="o">=</span> <span class="s2">"src/bin_client/test/contracts/opcodes/add_example.tz"</span>
<span class="n">run_script_res</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">run_script</span><span class="p">(</span>
<span class="n">add_contract</span><span class="p">,</span> <span class="s1">'None'</span><span class="p">,</span> <span class="s1">'Pair 10 15'</span><span class="p">,</span> <span class="n">amount</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">trace_stack</span><span class="o">=</span><span class="kc">True</span>
<span class="p">)</span>
<span class="k">assert</span> <span class="n">run_script_res</span><span class="o">.</span><span class="n">storage</span> <span class="o">==</span> <span class="s1">'(Some 25)'</span>
</code></pre></div>
<p>Using this method, we add regression testing to the existing test cases
simply by changing the fixture.</p>
<h3 id="scaling-down-the-pedantry-adding-output-conversions"><a class="toclink" href="#scaling-down-the-pedantry-adding-output-conversions">Scaling down the pedantry: adding output conversions</a></h3>
<p>One difficulty with regression testing is dealing with output that may
change from one test run to another. For instance, a contract that uses
the <code>NOW</code> instruction will have a different stack trace on each run,
as the timestamp that is pushed by <code>NOW</code> will differ on each execution.
Like a merciless river, time flows ever forwards.</p>
<p>There is little we can do to change the laws of physics, but we can
apply “output conversions” to the logs written by <code>pytest</code> to replace
such data in the output with fixed markers.</p>
<p>We have written a <code>client_regtest_scrubbed</code> fixture that extends
<code>client_regtest</code> with a series of such conversions. For example, a
timestamp such as <code>2019-09-23T10:59:00Z</code> is replaced by <code>[TIMESTAMP]</code>
and operation hashes with <code>[OPERATION_HASH]</code>.</p>
<h3 id="an-issue-with-long-paths-problem-in-pytest-regtest"><a class="toclink" href="#an-issue-with-long-paths-problem-in-pytest-regtest">An issue with long paths problem in <code>pytest-regtest</code></a></h3>
<p>During the implementation of regression testing, we discovered an
issue in <code>pytest-regtest</code> generation of log files. Tests with long
parameterizations would create log files with names whose length
exceeded the operating system path length limit. We have patched
<code>pytest-regtest</code> to replace explicit parameters in log file names with
a hash, if the name would exceeds the limits imposed by the operating system.</p>
<h3 id="discussion"><a class="toclink" href="#discussion">Discussion</a></h3>
<p>We have followed the method outlined above to add regression testing
to Tezos integration tests. Notably in the existing test suite
<a href="[merge request
1275](https://gitlab.com/tezos/tezos/merge_requests/1275/)"><code>test_contracts_opcode.py</code></a> that tests
Michelson instruction semantics. We have also applied it to test
<a href="https://gitlab.com/tezos/tezos/-/merge_requests/1530">macro
expansion</a>.</p>
<p>As a result, Michelson instructions are now regression test
enabled. This decreases the risk that any future change inadvertently
changes macro expansion, the semantics or the gas cost of any
instruction, as doing so will alert the developer by breaking the test
suite in many cases. However, due to the dynamic nature of gas costs,
it is impossible for any test suite to cover all behaviors. Think of
the addition of numbers, whose cost depends on the size of the
operands. Tests can only cover a finite number of operands, but since
integers and naturals are unbounded in Michelson, they can never cover
all. Consequently, a change that changes the gas cost for an untested
pair of operands will not be detected. Nonetheless, regression testing
provides an important security net that helps to nip bugs early in the
bud. Moving forward, we would like to extend regression testing to
other parts of the test suite.</p>
<p>Our current approach to output conversion is quite ad hoc. We use the
same set of output conversions for all tests. For examples, this
results in all timestamps being removed from the output, even those
that are constant between two runs. In the future, we would like to
use test-specific conversions targeting directly the parts of the
output that is expected to change.</p>
<p>Finally, regression testing is coarse-grained and so also
<strong>fragile</strong>. In addition to picking up unintended changes, regression
testing also detects intended, harmless changes.
For instance, in <a href="https://gitlab.com/metastatedev/tezos/-/merge_requests/54">merge request
metastatedev/tezos!54</a>,
the <code>UNPAIR</code> macro is replaced by an instruction. <code>UNPAIR</code> being
widely used, this change breaks all very large number of regression
tests. If such a change also modifies instruction semantics or gas usage in an unintended
way, then bugs may be introduced. In such cases, it is important that
the developer is very careful when inspecting the diffs of the
regression logs and that they keep the change to the code minimal
to ease review!</p>
<p>Indeed, as with any method for verification and validation, regression
testing should be approached as a complementary activity to careful
code review and formal methods.</p>Meanwhile at Nomadic Labs #62020-02-03T17:00:00+01:002020-02-03T17:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2020-02-03:/meanwhile-at-nomadic-labs-6.html<p>Summary:</p>
<p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>It’s time for yet another post in the “meanwhile at Nomadic Labs” series, and
we have a lot to talk about. Dive in to find out what you might have missed in January! </p>
<ul>
<li>
<p>Nomadic Labs partnered with <a href="https://software.imdea.org/">the <span class="caps">IMDEA</span> Software Institute</a>, a leading
research institute in Madrid. You …</p></li></ul><p>Summary:</p>
<p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>It’s time for yet another post in the “meanwhile at Nomadic Labs” series, and
we have a lot to talk about. Dive in to find out what you might have missed in January! </p>
<ul>
<li>
<p>Nomadic Labs partnered with <a href="https://software.imdea.org/">the <span class="caps">IMDEA</span> Software Institute</a>, a leading
research institute in Madrid. You can read more about the
partnership <a href="https://wp.software.imdea.org/tezos/">here</a> or in our <a href="https://www.linkedin.com/feed/update/urn:li:activity:6623979278000287744">press release</a>. </p>
</li>
<li>
<p>A new research program dedicated to Blockchain technology was founded in
partnership with <a href="https://www.inria.fr/en">Inria</a>, the French National
Institute for Research in Digital Science and Technology.</p>
</li>
</ul>
<p>These collaborations clearly demonstrate our focus to support more open
scientific research that can benefit Tezos and the entire Blockchain community.</p>
<ul>
<li>
<p>Alexandre Doussot presented “How to Develop Smart Contracts with Tezos” at the
<a href="https://p2p.paris/en/event/festival-0/">Paris <span class="caps">P2P</span> Festival</a>. This 1.5h
workshop gave an introduction on Michelson and Ligo, two fundamental languages
for the Tezos ecosystem.</p>
</li>
<li>
<p>Julien Tesson, Raphaël Cauderlier, Bruno Bernardo and Basile Pesin, a former intern at Nomadic Labs, published <a href="https://arxiv.org/abs/2001.02630">“Albert, an intermediate smart-contract language for the Tezos blockchain”</a>. Keep an
eye out for the <a href="http://fc20.ifca.ai/">Financial Cryptography and Data Security Conference 2020</a> in Malaysia on February 14th. Raphaël Cauderlier
will hold a presentation, giving more details on Albert during the
<a href="https://fc20.ifca.ai/wtsc/">4th Workshop on Trusted Smart Contracts</a>.</p>
</li>
<li>
<p>To celebrate our partnership, we organized a <a href="https://wp.software.imdea.org/tezos/events/">Tezos Blockchain Workshop</a> in cooperation with
<span class="caps">IMDEA</span> Software. Michel Mauny presented the distinguishing features of the technology
used within Tezos and the decentralized ecosystem supporting its development. </p>
</li>
<li>
<p>As promised, we published a second blog post on the The Spending Limit Contract: <a href="https://research-development.nomadic-labs.com/formally-verifying-a-critical-smart-contract.html">Formally Verifying a Critical Smart Contract</a>. This post goes into more detail on how we formally verified the Spending Limit Contract used in the Cortez mobile Tezos wallet. The application has been updated on the Google Play Store and we are working to get it to iOS as soon as possible. Android users can already head over to the <a href="https://play.google.com/store/apps/details?id=com.tezcore.cortez&hl=en">Play Store</a> and download the latest version of Cortez in order to start setting spending limits.</p>
</li>
<li>
<p>We started a series of dedicated hackathons in order to improve documentation
on the Michelson programming language. You can view a preview
<a href="https://michelson.nomadic-labs.com/">here</a></p>
</li>
</ul>
<p>Two smart contracts used in camlCase, which will serve as critical components
of the decentralized Dexter digital asset exchange, are currently being
formally verified. You can read more about Dexter in camlCase’s <a href="https://medium.com/@camlcase/dexter-27ae69ac79ed">blog
post</a>. </p>
<ul>
<li>
<p>Nomadic Labs visited <a href="https://concordium.com/team/#scientificTeam">Cobra</a>,
short for the Concordium Blockchain Research Center Aarhus. Due to their
similar interests in academic research, we discussed possible collaborations
and gave a technical Tezos presentation. </p>
</li>
<li>
<p>We attended The annual Symposium on Principles of Programming Languages (<a href="https://popl20.sigplan.org/"><span class="caps">POPL</span>
2020</a>) in New Orleans. Michel Mauny, Germán
Delbianco and Bruno Bernardo were present to answer Tezos-related questions.</p>
</li>
</ul>
<p>Nomadic Labs is proud to have sponsored the Southwestern Europe Regional programming
Contest (<a href="https://swerc.eu/2019/about/"><span class="caps">SWERC</span></a>) 2019-2020, which was organized
by the “Institut Polytechnique de Paris” and took place at Télécom Paris on
January 25th and 26th. Mehdi Bouaziz from Nomadic Labs was the judge during the
“Problem analysis session”. A few lucky winners went home with a brand new
ledger hardware wallet. You can view the video including the award ceremony
<a href="https://www.youtube.com/watch?v=G8bia4VEVXA&feature=youtu.be">here</a>.</p>
<ul>
<li>
<p>Pietro Abate from Nomadic Labs attended the European Blockchain Convention in Barcelona.</p>
</li>
<li>
<p>We are equally proud to be one of 11 founding members
of <a href="https://www.adan.eu/"><span class="caps">ADAN</span></a>, the “Association for the Development of
Digital Assets”. By bringing together French industry actors and representing
them, <span class="caps">ADAN</span> aims to promote the development of digital assets in France and Europe.</p>
</li>
<li>
<p>Last but not least, we published <a href="https://arxiv.org/abs/2001.11965">Tenderbake</a>, a classical <span class="caps">BFT</span> style consensus for public Blockchains. Inspired by Tendermint, Tenderbake is a first exercise in experimenting with deterministic finality in Tezos. Tenderbake is a collaboration with <a href="http://www-list.cea.fr/en/"><span class="caps">CEA</span> <span class="caps">LIST</span></a>.</p>
</li>
</ul>
<p>Stay tuned for more updates next month.</p>How to write a Tezos protocol - part 22020-01-23T18:00:00+01:002020-01-23T18:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2020-01-23:/how-to-write-a-tezos-protocol-part-2.html<p>This is the second post of a tutorial series on how to implement
a Tezos protocol.</p>
<p>In the <a href="https://research-development.nomadic-labs.com/how-to-write-a-tezos-protocol.html">first
post</a>, we
saw how to write, compile, register, activate and use an extremely simple
protocol. We also looked at the interface between the protocol and the
<em>shell</em>.</p>
<p>In this post, we consider a new protocol called <code>demo_counter</code> which extends
<code>demo_noops</code> from the first post in several ways.</p>
<ul>
<li>Blocks can contain simple operations, whose effects update the blockchain state.</li>
<li>It is parameterized by protocol parameters passed at activation time.</li>
<li>It defines <span class="caps">REST</span> services (a.k.a. RPCs), in addition to the generic ones
already available from the shell.</li>
<li>It defines a client library, extending <code>tezos-client</code> with protocol-specific commands.</li>
</ul>
<p>A large part of this post is devoted to the client library. While this library is not
part of the protocol per se, it is needed if we want to communicate
with the node in any meaningful way.</p>
<p>This protocol and the client library also make use of additional libraries,
such as command-line parsing tools, error monads, RPCs… Describing them
in detail would be beyond the scope of this post, but we try to provide
enough information to keep the post self-contained.</p>
<p><code>demo_counter</code> can be found in revision <code>254be33</code> of the <code>master</code> branch on the
Tezos repository. <code>demo_counter</code> is located in <code>src/proto_demo_counter/</code>.</p>
<p>We refer to the first post for compilation instructions. In most cases, it
should be enough to run</p>
<div class="highlight"><pre><span></span><code>make build-deps
make
</code></pre></div>
<h1 id="protocol"><a class="toclink" href="#protocol">Protocol</a></h1>
<p>The protocol is referred to by the hash
<code>ProtoDemoCounterDemoCounterDemoCounterDemoCou4LSpdT</code>. We can check it is
indeed known by the node.</p>
<div class="highlight"><pre><span></span><code># tezos-admin-client list protocols
ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK
ProtoDemoCounterDemoCounterDemoCounterDemoCou4LSpdT
ProtoDemoNoopsDemoNoopsDemoNoopsDemoNoopsDemo6XBoYp
ProtoGenesisGenesisGenesisGenesisGenesisGenesk612im
</code></pre></div>
<p>It is defined by several modules:</p>
<div class="highlight"><pre><span></span><code>ls src/proto_demo_counter/lib_protocol/*.mli
lib_protocol/apply.mli lib_protocol/main.mli lib_protocol/receipt.mli
lib_protocol/error.mli lib_protocol/proto_operation.mli lib_protocol/services.mli
lib_protocol/header.mli lib_protocol/proto_params.mli lib_protocol/state.mli
</code></pre></div>
<p>Most protocol-specific types required in <code>Main</code> are now
defined in separate modules.</p>
<ul>
<li><code>block_header_data</code> is defined as <code>Header.t</code>,</li>
<li><code>operation_receipt</code> is defined as <code>Receipt.t</code>,</li>
<li><code>operation_data</code> is defined as <code>Proto_operation.t</code>.</li>
</ul>
<p>As for <code>demo_noops</code>, <code>block_header_data</code> is still a string, and fitness
is defined as the height of the chain.
More interesting are the protocol operations and the operation receipt.</p>
<p>The protocol defines three operations in <code>Proto_operation.t</code>, which act on a
state <code>State.t</code> stored in the protocol
<em>context</em>. As seen in the first post, in the Tezos model each operation is
applied to a context and can produce a new context, the context is a map
that can be seen as the current state of the blockchain. For <code>proto_counter</code>,
the context maps a key <code>"state"</code> to a serialized form of <code>State.t</code>.</p>
<p><code>State</code> is simply a couple of nonnegative counters (which we can also view
as the balances of two accounts).</p>
<div class="highlight"><pre><span></span><code>type t = { a : int32; b : int32 }
</code></pre></div>
<p>Operations are defined in <code>Proto_operation</code> as</p>
<div class="highlight"><pre><span></span><code>type t =
| IncrA
| IncrB
| Transfer of Int32.t (* transfer from A to B, possibly a negative amount *)
</code></pre></div>
<p>The module <code>Apply</code> defines a function</p>
<div class="highlight"><pre><span></span><code>val<span class="w"> </span>apply<span class="w"> </span>:<span class="w"> </span>State.t<span class="w"> </span>-><span class="w"> </span>Proto_operation.t<span class="w"> </span>-><span class="w"> </span>State.t<span class="w"> </span>option
</code></pre></div>
<p>that applies the operation as expected. Some operations may be invalid (and in this case <code>apply</code> returns <code>None</code>).
For instance, transfer requires that both counters stay nonnegative, and
increment operations require that counters don’t overflow.</p>
<p>Operation application is defined by the function <code>Main.apply_operation</code>.</p>
<div class="highlight"><pre><span></span><code>val<span class="w"> </span>apply_operation<span class="w"> </span>:<span class="w"> </span>validation_state<span class="w"> </span>-><span class="w"> </span>operation<span class="w"> </span>->
<span class="w"> </span>(validation_state<span class="w"> </span>*<span class="w"> </span>operation_receipt)<span class="w"> </span>tzresult<span class="w"> </span>Lwt.t
let<span class="w"> </span>apply_operation<span class="w"> </span>validation_state<span class="w"> </span>operation<span class="w"> </span>=
<span class="w"> </span>Logging.log_notice<span class="w"> </span>"apply_operation";
<span class="w"> </span>let<span class="w"> </span>{<span class="w"> </span>context<span class="w"> </span>;<span class="w"> </span>fitness<span class="w"> </span>}<span class="w"> </span>=<span class="w"> </span>validation_state<span class="w"> </span>in
<span class="w"> </span>State.get_state<span class="w"> </span>context<span class="w"> </span>>>=<span class="w"> </span>fun<span class="w"> </span>state<span class="w"> </span>->
<span class="w"> </span>match<span class="w"> </span>Apply.apply<span class="w"> </span>state<span class="w"> </span>operation.protocol_data<span class="w"> </span>with
<span class="w"> </span>|<span class="w"> </span>None<span class="w"> </span>-><span class="w"> </span>Error_monad.fail<span class="w"> </span>Error.Invalid_operation
<span class="w"> </span>|<span class="w"> </span>Some<span class="w"> </span>state<span class="w"> </span>->
<span class="w"> </span>let<span class="w"> </span>receipt<span class="w"> </span>=<span class="w"> </span>Receipt.create<span class="w"> </span>"operation<span class="w"> </span>applied<span class="w"> </span>successfully"<span class="w"> </span>in
<span class="w"> </span>State.update_state<span class="w"> </span>context<span class="w"> </span>state<span class="w"> </span>>>=<span class="w"> </span>fun<span class="w"> </span>context<span class="w"> </span>->
<span class="w"> </span>return<span class="w"> </span>({<span class="w"> </span>context<span class="w"> </span>;<span class="w"> </span>fitness<span class="w"> </span>},<span class="w"> </span>receipt)
</code></pre></div>
<p>This is quite straightforward. If the application succeeds, fitness is left
unchanged and the resulting context contains the updated state. This function
also returns a <em>receipt</em> that describes the effect of the operation. In
this protocol, the receipt is simply a string, but it could be more
descriptive. If the application fails, an error is returned via an <a href="https://tezos.gitlab.io/developer/error_monad.html">error monad</a>. All protocol errors are
registered in <code>Error</code>.</p>
<h2 id="protocol-parameters"><a class="toclink" href="#protocol-parameters">Protocol parameters</a></h2>
<p>We saw in the first post that when a protocol is activated, we can
pass to it initialization parameters through a <span class="caps">JSON</span> value. This value
is provided by a user through a file argument,
e.g., <code>protocol_parameters.json</code>, to the activation command.
The <code>demo_noops</code> protocol did not take advantage of this feature, but <code>demo_counter</code> uses a
<span class="caps">JSON</span> value of the form <code>{'init_a': A, 'init_b': B}</code>, where <code>A</code> and <code>B</code>
are the initial values of the counters.</p>
<p>The type of the protocol parameters and their encoding are defined in
<code>Proto_params</code>.</p>
<p>By convention, the protocol parameters are stored in the context under
the key <code>"protocol_parameters"</code>. The activation operation of the
<code>genesis</code> protocol sets the parameters under this key, and they are
retrieved by <code>demo_counter</code> in <code>Main.init</code>.</p>
<h2 id="rpc-services"><a class="toclink" href="#rpc-services"><span class="caps">RPC</span> Services</a></h2>
<p>The protocol implements two services.</p>
<ul>
<li><code>/chains/main/blocks/head/counter/a</code> returns the value of counter <code>a</code></li>
<li><code>/chains/main/blocks/head/counter/b</code> returns the value of counter <code>b</code></li>
</ul>
<p>Services rely on the <code>RPC_*</code> modules accessible through the
protocol <a href="http://tezos.gitlab.io/developer/protocol_environment.html">environment</a>.
Ultimately, these modules are implemented by the <code>tezos-rpc</code> library.</p>
<p>Services are registered by the function <code>Services.rpc_services</code>,
which is called by <code>Main.rpc_services</code> at protocol activation.</p>
<div class="highlight"><pre><span></span><code>val<span class="w"> </span>rpc_services<span class="w"> </span>:<span class="w"> </span>Updater.rpc_context<span class="w"> </span>RPC_directory.t
let<span class="w"> </span>rpc_services<span class="w"> </span>=<span class="w"> </span>Services.rpc_services
</code></pre></div>
<h2 id="compilation"><a class="toclink" href="#compilation">Compilation</a></h2>
<p>Two libraries are compiled from the protocol code.</p>
<ul>
<li><code>tezos-protocol-demo-counter</code> is linked to the client library (see below),</li>
<li><code>tezos-embedded-protocol-demo-counter</code> is linked to the node (see
<code>src/bin_node/dune</code>).</li>
</ul>
<p>Recall from the first post that the protocol hash and modules are
given in <code>TEZOS_PROTOCOL</code>.</p>
<div class="highlight"><pre><span></span><code>> cat src/proto_demo_counter/lib_protocol/TEZOS_PROTOCOL
{
"hash": "ProtoDemoCounterDemoCounterDemoCounterDemoCou4LSpdT",
"modules": ["Error", "Proto_params", "Header", "State",
"Proto_operation", "Receipt", "Apply", "Services", "Main"]
}
</code></pre></div>
<p>The compilation steps are given in a generic <code>dune</code> file common to all
protocols, and a protocol-specific file <code>dune.inc</code> that can be generated from
the <code>TEZOS_PROTOCOL</code> file. We will discuss compilation at greater length in
the next blog post.</p>
<h1 id="client-library"><a class="toclink" href="#client-library">Client library</a></h1>
<p>The client and the node interact using RPCs. In theory, we could write a
client for <code>demo_counter</code> from scratch in any language, but it is convenient
to simply extend <code>tezos-client</code> with a protocol-specific client library.
Hence, we can keep using generic features of <code>tezos-client</code>, such as wallet
management, and simply add new commands specific to the new protocol.
Moreover, we can use well-tested OCaml libraries to conveniently call shell
RPCs, and we can access to the protocol code from the client, for instance
to access some of its datatypes or services.</p>
<p>For <code>demo_counter</code>, the client library is <code>tezos-client-demo-counter</code> and it is
defined in <code>src/proto_demo_counter/lib_client/</code>. It is linked to
<code>tezos-client</code> (see <code>bin_client/dune</code>). It is composed of four modules
<code>Client_proto_args</code>, <code>Client_proto_commands</code>, <code>Client_proto_main</code>, <code>Protocol_client_context</code>
which we will describe as we go.</p>
<p>We can check that <code>demo_counter</code> is indeed known to the client.</p>
<div class="highlight"><pre><span></span><code># tezos-admin-client list understood protocols
ProtoDemoCou
ProtoALphaAL
ProtoGenesis
</code></pre></div>
<p>Note that <code>Proto_demo_noops</code> isn’t in the list since it doesn’t have a client library.</p>
<h2 id="user-interface"><a class="toclink" href="#user-interface">User interface</a></h2>
<p><code>demo_counter</code> adds a few new commands.</p>
<div class="highlight"><pre><span></span><code>#<span class="w"> </span><span class="nv">tezos</span><span class="o">-</span><span class="nv">client</span><span class="w"> </span><span class="o">-</span><span class="nv">p</span><span class="w"> </span><span class="nv">ProtoDemoCou</span><span class="w"> </span><span class="nv">man</span>
...
<span class="nv">Commands</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">protocol</span><span class="w"> </span><span class="nv">Demo_counter</span>:
<span class="w"> </span><span class="nv">bake</span><span class="w"> </span><span class="o"><</span><span class="nv">message</span><span class="o">></span>
<span class="w"> </span><span class="nv">Bake</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">block</span>
<span class="w"> </span><span class="o"><</span><span class="nv">message</span><span class="o">></span>:<span class="w"> </span><span class="nv">message</span><span class="w"> </span><span class="nv">in</span><span class="w"> </span><span class="nv">block</span><span class="w"> </span><span class="nv">header</span>
<span class="w"> </span><span class="nv">incra</span>
<span class="w"> </span><span class="nv">Increment</span><span class="w"> </span><span class="nv">A</span>
<span class="w"> </span><span class="nv">incrb</span>
<span class="w"> </span><span class="nv">Increment</span><span class="w"> </span><span class="nv">B</span>
<span class="w"> </span><span class="nv">transfer</span><span class="w"> </span><span class="o"><</span><span class="nv">amount</span><span class="o">></span>
<span class="w"> </span><span class="nv">transfer</span><span class="w"> </span><span class="nv">from</span><span class="w"> </span><span class="nv">A</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">B</span>
<span class="w"> </span><span class="o"><</span><span class="nv">amount</span><span class="o">></span>:<span class="w"> </span><span class="nv">amount</span><span class="w"> </span><span class="nv">taken</span><span class="w"> </span><span class="nv">from</span><span class="w"> </span><span class="nv">A</span><span class="w"> </span><span class="nv">and</span><span class="w"> </span><span class="nv">given</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">B</span><span class="w"> </span><span class="ss">(</span><span class="nv">possibly</span><span class="w"> </span><span class="nv">negative</span><span class="ss">)</span>
<span class="w"> </span><span class="nv">get_a</span>
<span class="w"> </span><span class="nv">Get</span><span class="w"> </span><span class="nv">A</span><span class="w"> </span><span class="nv">counter</span>
<span class="w"> </span><span class="nv">get_b</span>
<span class="w"> </span><span class="nv">Get</span><span class="w"> </span><span class="nv">B</span><span class="w"> </span><span class="nv">counter</span>
...
</code></pre></div>
<h2 id="clientnode-interaction"><a class="toclink" href="#clientnode-interaction">Client/Node interaction</a></h2>
<p>Typically, the client library interacts with the node in two ways.</p>
<ul>
<li>using shell services (through the <code>tezos-shell-services</code> library),</li>
<li>using protocol services (through the protocol library, i.e.
<code>tezos-protocol-demo-counter</code>).</li>
</ul>
<p>Let see how to use these libraries in practice.</p>
<h3 id="library-tezos-protocol-demo-counter"><a class="toclink" href="#library-tezos-protocol-demo-counter">Library <code>tezos-protocol-demo-counter</code></a></h3>
<p>The library <code>tezos-protocol-demo-counter</code> contains the protocol code and
its environment. The modules of the protocol are grouped in a module
<code>Protocol</code>. The environment is accessible from module <code>Protocol.Environment</code>.</p>
<p>Typically, we want to use the datatypes defined in the protocol. For instance,
to build blocks, we need access to the type of operations and block header
data. We also want to access protocol-defined services. In our case,
we can get the counter values using the <em>client stub</em> <code>Services.get_counter</code>.</p>
<p>Remark that although it is feasible, the client code should not use
protocol functions who read or write the protocol context.</p>
<h3 id="library-tezos-shell-services"><a class="toclink" href="#library-tezos-shell-services">Library <code>tezos-shell-services</code></a></h3>
<p>This library defines client stubs to call shell <span class="caps">RPC</span> services. The
<code>demo_counter</code> client library uses <em>injection services</em> and <em>block services</em>.</p>
<p>As an example, consider the (slightly simplified) function
<code>Shell_services.Injection.block</code> defined in module <code>Injection_services</code> in
<code>lib_shell_services/</code>.</p>
<div class="highlight"><pre><span></span><code>val<span class="w"> </span>block:
<span class="w"> </span>#RPC_context.simple<span class="w"> </span>-><span class="w"> </span>MBytes.t<span class="w"> </span>-><span class="w"> </span>Operation.t<span class="w"> </span>list<span class="w"> </span>list
<span class="w"> </span>-><span class="w"> </span>Block_hash.t<span class="w"> </span>tzresult<span class="w"> </span>Lwt.t
</code></pre></div>
<ul>
<li><code>RPC_context.simple</code> is the <em><span class="caps">RPC</span> context</em>, whic identifies the server, deals
with the networking aspects of the call, and serializes the transmitted values.</li>
<li><code>Mbytes.t</code> contains the encoded block header.</li>
<li><code>Operation.t</code> is the generic shell operation type.</li>
<li><code>Block_hash.t</code> is the hash of the injected block.</li>
</ul>
<p>This function calls the service <code>/injection/block</code> and serializes
the parameters and the returned value as expected.</p>
<p>The client library uses two injection services.</p>
<ul>
<li><code>Shell_services.Injection.block</code> to call <code>/injection/block</code>,</li>
<li><code>Shell_services.Injection.operation</code> to call <code>/injection/operation</code>.</li>
</ul>
<p>The other family of services, block services, is defined as a functor
parameterized by a protocol type <code>Block_services.PROTO</code> defined in
<code>lib_shell_services</code>. This allows the stubs to deserialize the <span class="caps">JSON</span> values
returned by the services, and to return the actual protocol types to the client.</p>
<p>In <code>client_proto_commands.ml</code>, we instantiate the functor
<code>Block_services.Make</code> with the <code>Protocol</code> module (there are two occurrences because we need to provide a module for both the current and the next protocol).</p>
<div class="highlight"><pre><span></span><code><span class="k">module</span> <span class="n">Demo_block_services</span> = <span class="n">Block_services</span>.<span class="n">Make</span>(<span class="n">Protocol</span>)(<span class="n">Protocol</span>)
</code></pre></div>
<p>The client library uses the following stubs:</p>
<ul>
<li><code>Demo_block_services.hash</code> to call <code>/chains/main/blocks/head/hash</code></li>
<li><code>Demo_block_services.Mempool.pending_operations</code> to call <code>/chains/main/mempool/pending_operations</code></li>
<li><code>Demo_block_services.Helpers.Preapply.block</code> to call <code>/chains/main/blocks/head/helpers/preapply/block</code></li>
<li><code>Demo_block_services.Helpers.Preapply.operations</code> to call <code>/chains/main/blocks/head/helpers/preapply/operations/</code></li>
</ul>
<h2 id="commands-implementation"><a class="toclink" href="#commands-implementation">Commands implementation</a></h2>
<p>Commands are implemented with the help of several libraries.</p>
<ul>
<li><code>tezos-clic</code> is a command-line parsing library.</li>
<li><code>tezos-client-commands</code> provides the registration function for new commands.</li>
<li><code>client-base</code> defines notably the <code>Client_context.full</code> class, which contains
the client context (e.g. wallet, printing facilities, <span class="caps">RPC</span> context…).</li>
</ul>
<p>The commands’ syntax is defined in the modules <code>Client_proto_main</code> and
<code>Client_proto_args</code>. Commands are registered using the function
<code>Client_command.register</code>.</p>
<p>Commands behavior is implemented in module <code>Client_proto_commands</code>. A command implementation
may use a value of type <code>Client_context.full</code>, which is provided by
the registration function. Through this object, commands can access
the client and <span class="caps">RPC</span> contexts.</p>
<p>More precisely, our command implementations use a
<code>Protocol_client_context.full</code> object, which is a specialized version of
<code>Client_context.full</code>, defined in <code>Protocol_client_context</code>.</p>
<p>Let us have a closer look to the block baking command <code>Client_proto_commands.bake</code>.</p>
<div class="highlight"><pre><span></span><code><span class="n">let</span><span class="w"> </span><span class="n">bake</span><span class="w"> </span><span class="p">(</span><span class="n">cctxt</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="n">Protocol_client_context</span><span class="p">.</span><span class="k">full</span><span class="p">)</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="n">unit</span><span class="w"> </span><span class="n">tzresult</span><span class="w"> </span><span class="n">Lwt</span><span class="p">.</span><span class="n">t</span><span class="w"> </span><span class="o">=</span>
<span class="w"> </span><span class="n">Demo_block_services</span><span class="p">.</span><span class="n">Mempool</span><span class="p">.</span><span class="n">pending_operations</span><span class="w"> </span><span class="n">cctxt</span><span class="w"> </span><span class="p">()</span>
<span class="w"> </span><span class="o">>>=</span><span class="vm">?</span><span class="w"> </span><span class="n">fun</span><span class="w"> </span><span class="err">{</span><span class="w"> </span><span class="n">applied</span><span class="p">;</span><span class="w"> </span><span class="n">_</span><span class="err">}</span><span class="w"> </span><span class="o">-></span>
<span class="w"> </span><span class="n">let</span><span class="w"> </span><span class="n">operations</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">List</span><span class="p">.</span><span class="k">map</span><span class="w"> </span><span class="n">snd</span><span class="w"> </span><span class="n">applied</span><span class="w"> </span><span class="ow">in</span>
<span class="w"> </span><span class="n">let</span><span class="w"> </span><span class="n">block_header_data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Header</span><span class="p">.</span><span class="k">create</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="ow">in</span>
<span class="w"> </span><span class="n">Demo_block_services</span><span class="p">.</span><span class="n">Helpers</span><span class="p">.</span><span class="n">Preapply</span><span class="p">.</span><span class="n">block</span><span class="w"> </span><span class="n">cctxt</span><span class="w"> </span><span class="o">[</span><span class="n">operations</span><span class="o">]</span><span class="w"> </span><span class="o">~</span><span class="nl">protocol_data</span><span class="p">:</span><span class="n">block_header_data</span>
<span class="w"> </span><span class="o">>>=</span><span class="vm">?</span><span class="w"> </span><span class="n">fun</span><span class="w"> </span><span class="p">(</span><span class="n">shell</span><span class="p">,</span><span class="w"> </span><span class="n">preapply_result</span><span class="p">)</span><span class="w"> </span><span class="o">-></span>
<span class="w"> </span><span class="n">let</span><span class="w"> </span><span class="n">block_header_data_encoded</span><span class="w"> </span><span class="o">=</span>
<span class="w"> </span><span class="n">Data_encoding</span><span class="p">.</span><span class="nc">Binary</span><span class="p">.</span><span class="n">to_bytes_exn</span><span class="w"> </span><span class="n">Header</span><span class="p">.</span><span class="n">encoding</span><span class="w"> </span><span class="n">block_header_data</span><span class="w"> </span><span class="ow">in</span>
<span class="w"> </span><span class="n">let</span><span class="w"> </span><span class="n">header</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="n">Block_header</span><span class="p">.</span><span class="n">t</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="err">{</span><span class="w"> </span><span class="n">shell</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="n">protocol_data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">block_header_data_encoded</span><span class="w"> </span><span class="err">}</span><span class="w"> </span><span class="ow">in</span>
<span class="w"> </span><span class="n">let</span><span class="w"> </span><span class="n">header_encoded</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Data_encoding</span><span class="p">.</span><span class="nc">Binary</span><span class="p">.</span><span class="n">to_bytes_exn</span><span class="w"> </span><span class="n">Block_header</span><span class="p">.</span><span class="n">encoding</span><span class="w"> </span><span class="n">header</span><span class="w"> </span><span class="ow">in</span>
<span class="w"> </span><span class="n">let</span><span class="w"> </span><span class="n">preapply_result</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">List</span><span class="p">.</span><span class="n">hd</span><span class="w"> </span><span class="n">preapply_result</span><span class="w"> </span><span class="ow">in</span>
<span class="w"> </span><span class="n">let</span><span class="w"> </span><span class="n">operations</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">[</span><span class="n">List.map snd preapply_result.applied</span><span class="o">]</span><span class="w"> </span><span class="ow">in</span>
<span class="w"> </span><span class="n">Shell_services</span><span class="p">.</span><span class="n">Injection</span><span class="p">.</span><span class="n">block</span><span class="w"> </span><span class="n">cctxt</span><span class="w"> </span><span class="n">header_encoded</span><span class="w"> </span><span class="n">operations</span>
<span class="w"> </span><span class="o">>>=</span><span class="vm">?</span><span class="w"> </span><span class="n">fun</span><span class="w"> </span><span class="n">block_hash</span><span class="w"> </span><span class="o">-></span>
<span class="w"> </span><span class="n">cctxt#message</span><span class="w"> </span><span class="ss">"Injected block %a"</span><span class="w"> </span><span class="n">Block_hash</span><span class="p">.</span><span class="n">pp_short</span><span class="w"> </span><span class="n">block_hash</span>
<span class="w"> </span><span class="o">>>=</span><span class="w"> </span><span class="n">fun</span><span class="w"> </span><span class="p">()</span><span class="w"> </span><span class="o">-></span>
<span class="w"> </span><span class="n">return_unit</span>
</code></pre></div>
<p>First, it retrieves the applied operations from the mempool using
<code>Demo_block_services.Mempool.pending_operations</code>. It then uses
the pre-apply service and ask the node to build a block based on
the proposed operations and block header data. The block is then
encoded and sent to the node through the injection service.</p>
<h1 id="sample-execution-of-the-protocol"><a class="toclink" href="#sample-execution-of-the-protocol">Sample execution of the protocol</a></h1>
<p>First we activate the protocol using <code>protocol_parameters.json</code> defined as</p>
<div class="highlight"><pre><span></span><code>{'init_a': 100, 'init_b': 100}
</code></pre></div>
<div class="highlight"><pre><span></span><code># tezos-client -block genesis activate protocol ProtoDemoCounterDemoCounterDemoCounterDemoCou4LSpdT with fitness 1 and key activator and parameters protocol_parameters.json --timestamp 2019-07-05T14:30:35Z
Injected BLf2cXRZKsby
</code></pre></div>
<p>This bakes block of level <code>1</code>, running protocol <code>genesis</code>, with <code>demo_counter</code>
scheduled for the next block.</p>
<div class="highlight"><pre><span></span><code># tezos-client rpc get /chains/main/blocks/head/metadata
{ "protocol": "ProtoGenesisGenesisGenesisGenesisGenesisGenesk612im",
"next_protocol": "ProtoDemoCounterDemoCounterDemoCounterDemoCou4LSpdT",
"test_chain_status": { "status": "not_running" }, "max_operations_ttl": 0,
"max_operation_data_length": 100, "max_block_header_length": 100,
"max_operation_list_length": [ { "max_size": 1000 } ] }
</code></pre></div>
<p>Although the head is a <code>genesis</code> block, <code>demo_counter</code> has already been
activated and we can bake an empty block using the <code>bake</code> command
from <code>demo_counter</code> client library.</p>
<div class="highlight"><pre><span></span><code># tezos-client bake '"This is block 2"'
Injected block BLrQqbn13Vrb
</code></pre></div>
<p>We can check that the block was baked properly, in particular
the block header data has been set as expected. We can also
see the protocol state <code>State.t</code> in the block metadata encoded
as specified by <code>State.encoding</code>.</p>
<div class="highlight"><pre><span></span><code># tezos-client rpc get /chains/main/blocks/head/
{ "protocol": "ProtoDemoCounterDemoCounterDemoCounterDemoCou4LSpdT",
"chain_id": "NetXdQprcVkpaWU",
"hash": "BLrQqbn13VrbzUprxQypzAg6fc7YmsHaZvGwGrHJk8a4eG6e11B",
"header":
{ "level": 2, "proto": 1,
"predecessor": "BLf2cXRZKsbygWJdtf1PBbrSg8yHkNK39bgoApvdbYBd1EX9ung",
"timestamp": "2019-07-05T14:30:36Z", "validation_pass": 1,
"operations_hash":
"LLoaGLRPRx3Zf8kB4ACtgku8F4feeBiskeb41J1ciwfcXB3KzHKXc",
"fitness": [ "01", "0000000000000002" ],
"context": "CoVpDgKDiWZ9xcodUFng1C8oGvfXEqBCD5XxQjB8Jrwptkx3vHUB",
"demo_block_header_data": "This is block 2" },
"metadata":
{ "protocol": "ProtoDemoCounterDemoCounterDemoCounterDemoCou4LSpdT",
"next_protocol": "ProtoDemoCounterDemoCounterDemoCounterDemoCou4LSpdT",
"test_chain_status": { "status": "not_running" },
"max_operations_ttl": 0, "max_operation_data_length": 100,
"max_block_header_length": 100,
"max_operation_list_length": [ { "max_size": 1000 } ], "demo_a": 100,
"demo_b": 100 }, "operations": [ [] ] }
</code></pre></div>
<p>We now inject three operations using client commands.</p>
<div class="highlight"><pre><span></span><code>tezos-client incra
Operation receipt: operation applied successfully
Injected: op5gBsE7EMi7
</code></pre></div>
<div class="highlight"><pre><span></span><code># tezos-client incrb
Operation receipt: operation applied successfully
Injected: oo2YhBbAY8Vr
</code></pre></div>
<div class="highlight"><pre><span></span><code># tezos-client transfer 10
Operation receipt: operation applied successfully
Injected: opJFLuHR98tf
</code></pre></div>
<p>The operations are known to the node, they appear as <em>applied</em>
in the node mempool.</p>
<div class="highlight"><pre><span></span><code># tezos-client rpc get /chains/main/mempool/pending_operations
{ "applied":
[ { "hash": "op45sL79jASRf41kpL5NDDbAUnQeTfwgZpVnZi1sXy4Cj5x18m9",
"branch": "BLa7SnHxjHqPTsGSE2fi8sHBm39u9g6Psd9qPZm4rJCqhzHdkSp",
"IncrA": {} },
{ "hash": "opV6ZMR2z2ZZUSjetzPTknPisjN6x5eFCQnAWhuKNic6GsiRLW7",
"branch": "BLa7SnHxjHqPTsGSE2fi8sHBm39u9g6Psd9qPZm4rJCqhzHdkSp",
"IncrB": {} },
{ "hash": "oo1a3gwKnXFqaHuhpgMb5x69wm3mwbid4hn8Ry1iX8jbvXzQQs7",
"branch": "BLa7SnHxjHqPTsGSE2fi8sHBm39u9g6Psd9qPZm4rJCqhzHdkSp",
"Transfer": 10 } ], "refused": [], "branch_refused": [],
"branch_delayed": [], "unprocessed": [] }
</code></pre></div>
<p>We bake the third block.</p>
<div class="highlight"><pre><span></span><code>tezos-client bake '"This is block 3"'
Injected block BLz4SrcTnBQU
</code></pre></div>
<p>We can see now that the three operations appear in the <code>operations</code>
section of the block, encoded as specified by <code>Proto_operation.encoding</code>.
The receipt for each operation also appears in this section.</p>
<div class="highlight"><pre><span></span><code>tezos-client rpc get /chains/main/blocks/head/
{ "protocol": "ProtoDemoCounterDemoCounterDemoCounterDemoCou4LSpdT",
"chain_id": "NetXdQprcVkpaWU",
"hash": "BLz4SrcTnBQUiXXks1FTzGR9d5MsX6F1mhZ2g23bHTcaQwJbk3S",
"header":
{ "level": 3, "proto": 1,
"predecessor": "BLrQqbn13VrbzUprxQypzAg6fc7YmsHaZvGwGrHJk8a4eG6e11B",
"timestamp": "2019-07-05T14:30:38Z", "validation_pass": 1,
"operations_hash":
"LLoZctr62cmk2pvVu2dqX5nv8rA7PHi3xRNGe6mbqmtAQPKtwHuKK",
"fitness": [ "01", "0000000000000003" ],
"context": "CoVXzytYqZcw4RQknAZJpK3RAFeLrzcZG2zDMdzuacpDPjX7YSor",
"demo_block_header_data": "This is block 3" },
"metadata":
{ "protocol": "ProtoDemoCounterDemoCounterDemoCounterDemoCou4LSpdT",
"next_protocol": "ProtoDemoCounterDemoCounterDemoCounterDemoCou4LSpdT",
"test_chain_status": { "status": "not_running" },
"max_operations_ttl": 0, "max_operation_data_length": 100,
"max_block_header_length": 100,
"max_operation_list_length": [ { "max_size": 1000 } ], "demo_a": 91,
"demo_b": 111 },
"operations":
[ [ { "protocol": "ProtoDemoCounterDemoCounterDemoCounterDemoCou4LSpdT",
"chain_id": "NetXdQprcVkpaWU",
"hash": "op5gBsE7EMi7gsR3xtSMMQms9XN8Pka5N1pT8XGuN1iP2siizkx",
"branch": "BLrQqbn13VrbzUprxQypzAg6fc7YmsHaZvGwGrHJk8a4eG6e11B",
"data": { "IncrA": {} },
"receipt":
{ "demo_operation_receipt": "operation applied successfully" } },
{ "protocol": "ProtoDemoCounterDemoCounterDemoCounterDemoCou4LSpdT",
"chain_id": "NetXdQprcVkpaWU",
"hash": "oo2YhBbAY8Vr2ASXj5k3PggXoxDVYWAQGvK1Sm22GhPFcXdvjQq",
"branch": "BLrQqbn13VrbzUprxQypzAg6fc7YmsHaZvGwGrHJk8a4eG6e11B",
"data": { "IncrB": {} },
"receipt":
{ "demo_operation_receipt": "operation applied successfully" } },
{ "protocol": "ProtoDemoCounterDemoCounterDemoCounterDemoCou4LSpdT",
"chain_id": "NetXdQprcVkpaWU",
"hash": "opJFLuHR98tfTTXGhsnMMjg6KxgVW8qx7LZ5q2nrhoySXySqZSS",
"branch": "BLrQqbn13VrbzUprxQypzAg6fc7YmsHaZvGwGrHJk8a4eG6e11B",
"data": { "Transfer": 10 },
"receipt":
{ "demo_operation_receipt": "operation applied successfully" } } ] ] }
</code></pre></div>
<p>We can finally test our two RPCs to query the counter values.</p>
<div class="highlight"><pre><span></span><code>tezos-client rpc get /chains/main/blocks/head/counter/a
91
</code></pre></div>
<div class="highlight"><pre><span></span><code>tezos-client rpc get /chains/main/blocks/head/counter/b
111
</code></pre></div>
<p>The node’s trace is similar to the one presented in the previous blog post. What we see in addition are three chunks of output of the form:</p>
<div class="highlight"><pre><span></span><code><span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">20</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">counter</span><span class="p">:</span><span class="w"> </span><span class="n">begin_construction</span><span class="w"> </span><span class="p">(</span><span class="n">mempool</span><span class="p">):</span><span class="w"> </span><span class="n">pred_fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000002</span><span class="w"> </span><span class="n">constructed</span><span class="w"> </span><span class="n">fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000003</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">20</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">counter</span><span class="p">:</span><span class="w"> </span><span class="n">apply_operation</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">20</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">counter</span><span class="p">:</span><span class="w"> </span><span class="n">finalize_block</span><span class="p">:</span><span class="w"> </span><span class="n">fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000003</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">20</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">counter</span><span class="p">:</span><span class="w"> </span><span class="n">apply_operation</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">20</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">counter</span><span class="p">:</span><span class="w"> </span><span class="n">apply_operation</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">20</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">prevalidator</span><span class="o">.</span><span class="n">NetXdQprcVkpa</span><span class="o">.</span><span class="n">ProtoDemoCou_1</span><span class="p">:</span><span class="w"> </span><span class="n">injecting</span><span class="w"> </span><span class="n">operation</span><span class="w"> </span><span class="n">op25ATABm3GS2AxZr8QFAaz9qor6JSqjYuYx4MSVXYtVZ86LaxF</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">20</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">prevalidator</span><span class="o">.</span><span class="n">NetXdQprcVkpa</span><span class="o">.</span><span class="n">ProtoDemoCou_1</span><span class="p">:</span><span class="w"> </span><span class="n">Pushed</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">08</span><span class="n">T12</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">20</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">,</span><span class="w"> </span><span class="n">Treated</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">08</span><span class="n">T12</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">20</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">,</span><span class="w"> </span><span class="n">Completed</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">08</span><span class="n">T12</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">20</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span>
</code></pre></div>
<p>The first 3 lines correspond to the call <code>Demo_block_services.Helpers.Preapply.operations cctxt [op]</code> and the next four lines to the call <code>Shell_services.Injection.operation</code> (both calls triggered by <code>Client_proto_commands.inject_op</code>).
Finally, when the last block is created, we see the following output:</p>
<div class="highlight"><pre><span></span><code><span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">22</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">counter</span><span class="p">:</span><span class="w"> </span><span class="n">begin_construction</span><span class="w"> </span><span class="p">(</span><span class="n">block</span><span class="p">):</span><span class="w"> </span><span class="n">pred_fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000002</span><span class="w"> </span><span class="n">constructed</span><span class="w"> </span><span class="n">fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000003</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">22</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">counter</span><span class="p">:</span><span class="w"> </span><span class="n">apply_operation</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">22</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">counter</span><span class="p">:</span><span class="w"> </span><span class="n">apply_operation</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">22</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">counter</span><span class="p">:</span><span class="w"> </span><span class="n">apply_operation</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">22</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">counter</span><span class="p">:</span><span class="w"> </span><span class="n">finalize_block</span><span class="p">:</span><span class="w"> </span><span class="n">fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000003</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">22</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">counter</span><span class="p">:</span><span class="w"> </span><span class="n">begin_application</span><span class="p">:</span><span class="w"> </span><span class="n">pred_fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000002</span><span class="w"> </span><span class="n">block_fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000003</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">22</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">counter</span><span class="p">:</span><span class="w"> </span><span class="n">apply_operation</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">22</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">counter</span><span class="p">:</span><span class="w"> </span><span class="n">apply_operation</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">22</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">counter</span><span class="p">:</span><span class="w"> </span><span class="n">apply_operation</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">22</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">counter</span><span class="p">:</span><span class="w"> </span><span class="n">finalize_block</span><span class="p">:</span><span class="w"> </span><span class="n">fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000003</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">22</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">validator</span><span class="o">.</span><span class="n">block</span><span class="p">:</span><span class="w"> </span><span class="n">Block</span><span class="w"> </span><span class="n">BMQFowYxF9WS6FjVRL2hMDEcV3LrViLFe9v4yo4W9tSPcyUq7Md</span><span class="w"> </span><span class="n">successfully</span><span class="w"> </span><span class="n">validated</span>
<span class="n">Jul</span><span class="w"> </span><span class="mi">8</span><span class="w"> </span><span class="mi">14</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">22</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">validator</span><span class="o">.</span><span class="n">block</span><span class="p">:</span><span class="w"> </span><span class="n">Pushed</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">08</span><span class="n">T12</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">22</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">,</span><span class="w"> </span><span class="n">Treated</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">08</span><span class="n">T12</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">22</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">,</span><span class="w"> </span><span class="n">Completed</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">07</span><span class="o">-</span><span class="mi">08</span><span class="n">T12</span><span class="p">:</span><span class="mi">27</span><span class="p">:</span><span class="mi">22</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span>
</code></pre></div>
<p>Here again, the chunks can be seen as being composed of two parts, one for <code>Demo_block_services.Helpers.Preapply.block</code> and one for <code>Shell_services.Injection.block</code>. In each of the parts <code>apply_operation</code> is called three times, once for each operation included in the block.</p>
<p>This scenario can be reproduced using the following python test
(this requires to install Python tools and libraries, as described
<a href="http://tezos.gitlab.io/developer/python_testing_framework.html">here</a>. It
launches a node, and runs the client commands to activate the protocol
and interact with it.</p>
<div class="highlight"><pre><span></span><code>cd tests_python
mkdir tmp
pytest -s tests/test_proto_demo_counter.py --log-dir=tmp
</code></pre></div>
<p>The node trace appears in <code>tmp/node0_0.txt</code>.</p>
<h1 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h1>
<p>We presented a simple protocol <code>demo_counter</code> which explores further the
interface between the shell and the protocol, and uses more features available
to the protocol developer such as <span class="caps">RPC</span> services. Besides, this protocol comes
with a library that extends <code>tezos-client</code> with new commands to
interact with the protocol.</p>
<p>In the next post, we will present <code>demo_account</code>, an account-based protocol,
where transactions and blocks must be signed in order to be valid.</p>Formally Verifying a Critical Smart Contract2020-01-20T18:00:00+01:002020-01-20T18:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2020-01-20:/formally-verifying-a-critical-smart-contract.html<p>We present the formal verification of the Spending Limit Contract, a critical component of the Cortez wallet.</p><p align="center" style="margin: 40px 0">
<img style="margin-right: 60px" src="images/slc-verif/cortez.png" />
<img src="images/slc-verif/mi-cho-coq.png" />
</p>
<p>One of the main goals of <a href="https://nomadic-labs.com/">Nomadic Labs</a> is
the development and applications of
<a href="https://en.wikipedia.org/wiki/Formal_methods">formal methods</a> in the
domain of distributed software, blockchains and smart contracts.
In particular for the <a href="https://tezos.com/">Tezos</a> blockchain, for which we
also develop the Cortez smartphone wallet
(<a href="https://gitlab.com/nomadic-labs/cortez-android">Android</a>,
<a href="https://gitlab.com/nomadic-labs/cortez-ios">iPhone</a>).
This wallet helps Tezos users manage their account and funds in a safe and secure manner. </p>
<p>How can the user be assured that this critical software has been
correctly implemented? In this blog post, we detail how we have
applied formal methods to verify the safety of a core component of the
Cortez safety layer: <strong>the Spending Limit Contract smart contract</strong>.</p>
<p>We first detail the <a href="#the-spending-limit-contract">contract and its specification</a>. Even though the
deployed contract is hand-written in
<a href="http://tezos.gitlab.io/whitedoc/michelson.html">Michelson</a>, we here
give a more readable pseudocode in <a href="http://ligolang.org/">CameLIGO</a>. We then
overview <a href="#approaches-to-formal-verification">common approaches to formal methods</a>, before <a href="#mi-cho-coq">discussing the
tool we have used: Mi-Cho-Coq</a>. Finally, we discuss the <a href="#verifying-spending-limit-in-mi-cho-coq">verification
effort itself</a>.</p>
<h2 id="the-spending-limit-contract"><a class="toclink" href="#the-spending-limit-contract">The Spending Limit Contract</a></h2>
<p>The <strong>Spending Limit Contract</strong> (<span class="caps">SLC</span>) is a critical component of the safety layer of the
Cortez wallet,
<a href="https://research-development.nomadic-labs.com/cortez-security-by-using-the-spending-limit-contract.html">as discussed earlier on this blog</a>.
This contract ensures that only a limited amount of funds can be spent
within a given time window. This adds a layer of security to the
user’s account in the unfortunate event that their device is
stolen.
The contract accounts user spending in a queue of past transfers. This
queue is implemented using a pair of lists, a technique for
optimizing gas consumption<sup id="fnref:3"><a class="footnote-ref" href="#fn:3">3</a></sup>. However, this implementation increases the complexity
of the contract’s logic significantly.</p>
<p>The Spending Limit Contract specifies two roles: an
<em>administrator</em> and a <em>user</em>. They are identified by their respective
<code>key_hash</code>, and their calls to the contract must be authenticated through a
cryptographic signature using this <code>key_hash</code>.</p>
<p>Historically, the contract used <em>authentication by caller</em>. That is,
calls to protected entrypoint would be rejected if the caller’s
address did not correspond to that of the appropriate role. However,
this approach required the revelation of two public keys when deploying the
contract, as well as another revelation if either of the key should
be changed. As <a href="http://tezos.gitlab.io/introduction/howtouse.html#transfers-and-receipts">revelations are costly</a>,
the current version of <span class="caps">SLC</span> has
passed to <em>authentication by signature</em>. In this mode, the keys for
signing are not required to be originated. Additionally, the same
address can be used to call both administrator and user entrypoints,
even if their signing keys change.</p>
<p>The administrator has full control over the contract through the
<code>administer</code> entrypoint. First, they can reset its storage. Second,
they can pass it a <code>lambda</code> that produces a list of operations when
executed, that the contract then returns. For example, the first
feature can be used to change the identity of the user or the
administrator, or to modify the duration of the time window and the spending limit. The
second feature can be used to remove an arbitrary amount of tokens from the account.</p>
<p>In earlier version of the contract, the administrator was limited to
resetting the storage. Consequently, to perform a transfer larger than
the limit, the administrator would have to perform three calls: one to
raise the limit, a second to perform the transfer, and a third to
restore the limit. By letting the administrator pass a lambda, the
contract is both more flexible and less costly to use.</p>
<p>On the other hand, the manipulations of the user, passing through the
<code>transfer</code> entrypoint, is subject to the restrictions of the time
window and the spending limit. The user supplies a list of
transfers. The smart contract verifies that the total sum of these
transfers, added to any previous transfers in the time window,
does not exceed the limit, in which case they are executed.
Historically, the contract allowed only one transfer per call to
<code>transfer</code>. To amend this limitation, the current version allows the user to pass a list
of transfers, thus reducing the cost of performing multiple transfers.</p>
<p>The contract is composed of the <code>administer</code> and <code>transfer</code>
entrypoints, along with a simple unauthenticated entrypoint <code>receive</code>
used to receive funds. The actual contract is hand-written in
Michelson, but for a legible illustration, we give its pseudocode in
<a href="https://ligolang.org/">CameLIGO</a>. Consequently, the main function of
the contract dispatches between the entrypoints with their respective parameters:</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="n">param</span> <span class="o">=</span>
<span class="nc">Receive</span> <span class="k">of</span> <span class="kt">unit</span>
<span class="o">|</span> <span class="nc">Administer</span> <span class="k">of</span> <span class="n">administer_param</span>
<span class="o">|</span> <span class="nc">Transfer</span> <span class="k">of</span> <span class="n">transfer_param</span>
<span class="k">let</span> <span class="n">main</span> <span class="o">(</span><span class="n">param</span><span class="o">,</span> <span class="n">s</span> <span class="o">:</span> <span class="n">param</span> <span class="o">*</span> <span class="n">storage</span><span class="o">)</span> <span class="o">=</span>
<span class="k">match</span> <span class="n">param</span> <span class="k">with</span>
<span class="o">|</span> <span class="nc">Receive</span> <span class="n">p</span> <span class="o">-></span> <span class="n">receive</span> <span class="o">(</span><span class="n">p</span><span class="o">,</span> <span class="n">s</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">Administer</span> <span class="n">p</span> <span class="o">-></span> <span class="n">administer</span> <span class="o">(</span><span class="n">p</span><span class="o">,</span> <span class="n">s</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">Transfer</span> <span class="n">p</span> <span class="o">-></span> <span class="n">transfer</span> <span class="o">(</span><span class="n">p</span><span class="o">,</span> <span class="n">s</span><span class="o">)</span>
</code></pre></div>
<p>The full CameLIGO version, including the definition of <code>administrate_param</code> and <code>transfer_param</code>, can be found
<a href="files/slc-verif/SLC.mligo">here</a>. We now give an informal
specification of the desired behavior of each entrypoint.</p>
<h3 id="administer"><a class="toclink" href="#administer">Administer</a></h3>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="n">change_storage</span> <span class="o">(</span><span class="n">csp</span> <span class="o">,</span> <span class="n">sto</span> <span class="o">:</span> <span class="n">change_storage_param</span> <span class="o">*</span> <span class="n">storage</span><span class="o">)</span> <span class="o">:</span> <span class="n">operation</span> <span class="kt">list</span> <span class="o">*</span> <span class="n">storage</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">key</span> <span class="o">=</span> <span class="n">csp</span><span class="o">.</span><span class="mi">0</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">sign</span> <span class="o">=</span> <span class="n">csp</span><span class="o">.</span><span class="mi">1</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">new_sto</span> <span class="o">=</span> <span class="n">csp</span><span class="o">.</span><span class="mi">2</span> <span class="k">in</span>
<span class="k">if</span> <span class="nn">Crypto</span><span class="p">.</span><span class="n">hash_key</span> <span class="n">key</span> <span class="o">=</span> <span class="n">sto</span><span class="o">.</span><span class="n">key_administrator</span>
<span class="o">&&</span> <span class="nn">Crypto</span><span class="p">.</span><span class="n">check</span> <span class="n">key</span> <span class="n">sign</span> <span class="o">(</span><span class="nn">Bytes</span><span class="p">.</span><span class="n">pack</span> <span class="o">(</span><span class="n">new_sto</span><span class="o">,</span><span class="n">sto</span><span class="o">.</span><span class="n">salt_administrator</span><span class="o">))</span>
<span class="k">then</span>
<span class="o">((</span><span class="bp">[]</span> <span class="o">:</span> <span class="n">operation</span> <span class="kt">list</span><span class="o">),</span> <span class="n">new_sto</span><span class="o">)</span>
<span class="k">else</span>
<span class="o">(</span><span class="n">failwith</span> <span class="s2">"Bad signature"</span> <span class="o">:</span> <span class="n">operation</span> <span class="kt">list</span> <span class="o">*</span> <span class="n">storage</span><span class="o">)</span>
<span class="k">let</span> <span class="n">run_function</span> <span class="o">(</span><span class="n">rfp</span><span class="o">,</span> <span class="n">sto</span> <span class="o">:</span> <span class="n">run_function_param</span> <span class="o">*</span> <span class="n">storage</span> <span class="o">)</span> <span class="o">:</span> <span class="n">operation</span> <span class="kt">list</span> <span class="o">*</span> <span class="n">storage</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">key</span> <span class="o">=</span> <span class="n">rfp</span><span class="o">.</span><span class="mi">0</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">sign</span> <span class="o">=</span> <span class="n">rfp</span><span class="o">.</span><span class="mi">1</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">admin_function</span> <span class="o">=</span> <span class="n">rfp</span><span class="o">.</span><span class="mi">2</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">kh</span> <span class="o">=</span> <span class="n">rfp</span><span class="o">.</span><span class="mi">3</span> <span class="k">in</span>
<span class="k">if</span> <span class="nn">Crypto</span><span class="p">.</span><span class="n">hash_key</span> <span class="n">key</span> <span class="o">=</span> <span class="n">sto</span><span class="o">.</span><span class="n">key_administrator</span> <span class="o">&&</span>
<span class="nn">Crypto</span><span class="p">.</span><span class="n">check</span> <span class="n">key</span> <span class="n">sign</span> <span class="o">(</span><span class="nn">Bytes</span><span class="p">.</span><span class="n">pack</span> <span class="o">(</span><span class="n">admin_function</span><span class="o">,</span><span class="n">sto</span><span class="o">.</span><span class="n">salt_administrator</span><span class="o">))</span> <span class="k">then</span>
<span class="o">(</span><span class="n">admin_function</span> <span class="bp">()</span><span class="o">,</span>
<span class="o">{</span> <span class="n">key_user</span> <span class="o">=</span> <span class="n">sto</span><span class="o">.</span><span class="n">key_user</span> <span class="o">;</span>
<span class="n">key_administrator</span> <span class="o">=</span> <span class="n">kh</span> <span class="o">;</span>
<span class="n">current_threshold</span> <span class="o">=</span> <span class="n">sto</span><span class="o">.</span><span class="n">current_threshold</span><span class="o">;</span>
<span class="n">time_window</span> <span class="o">=</span> <span class="n">sto</span><span class="o">.</span><span class="n">time_window</span><span class="o">;</span>
<span class="n">queue</span> <span class="o">=</span> <span class="n">sto</span><span class="o">.</span><span class="n">queue</span> <span class="o">;</span>
<span class="n">salt_user</span> <span class="o">=</span> <span class="n">sto</span><span class="o">.</span><span class="n">salt_user</span> <span class="o">;</span>
<span class="n">salt_administrator</span> <span class="o">=</span> <span class="n">sto</span><span class="o">.</span><span class="n">salt_administrator</span> <span class="o">+</span> <span class="mi">1</span><span class="n">n</span><span class="o">})</span>
<span class="k">else</span>
<span class="o">(</span><span class="n">failwith</span> <span class="s2">"Bad signature"</span> <span class="o">:</span> <span class="n">operation</span> <span class="kt">list</span> <span class="o">*</span> <span class="n">storage</span><span class="o">)</span>
<span class="k">let</span> <span class="n">administer</span> <span class="o">(</span><span class="n">p</span><span class="o">,</span> <span class="n">s</span> <span class="o">:</span> <span class="n">administer_param</span> <span class="o">*</span> <span class="n">storage</span><span class="o">)</span> <span class="o">=</span>
<span class="k">match</span> <span class="n">p</span> <span class="k">with</span>
<span class="o">|</span> <span class="nc">Change_storage</span> <span class="n">cs</span> <span class="o">-></span> <span class="n">change_storage</span> <span class="o">(</span><span class="n">cs</span><span class="o">,</span> <span class="n">s</span><span class="o">)</span>
<span class="o">|</span> <span class="nc">Run_function</span> <span class="n">mt</span> <span class="o">-></span> <span class="n">run_function</span> <span class="o">(</span><span class="n">mt</span><span class="o">,</span> <span class="n">s</span><span class="o">)</span>
</code></pre></div>
<p>The <code>administer</code> entrypoint, illustrated by the above CameLIGO pseudocode,
receives a public key, a payload and a signature of this payload. It
verifies that the hash of this key equals the stored key hash
of the administrator. It also verifies that the signature of the
payload is valid and created by the administrator. If either of these
conditions are unfulfilled, the contract fails.</p>
<p>Then, depending on the payload, it either updates the storage of the
context, or executes the received <code>lambda</code> and returns the obtained
list of operations.</p>
<h3 id="transfer"><a class="toclink" href="#transfer">Transfer</a></h3>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="n">reverse</span> <span class="o">(</span><span class="n">l</span> <span class="o">:</span> <span class="o">(</span><span class="n">timestamp</span> <span class="o">*</span> <span class="n">tez</span><span class="o">)</span> <span class="kt">list</span><span class="o">)</span> <span class="o">:</span> <span class="o">(</span><span class="n">timestamp</span> <span class="o">*</span> <span class="n">tez</span><span class="o">)</span> <span class="kt">list</span> <span class="o">=</span>
<span class="nn">List</span><span class="p">.</span><span class="n">fold</span>
<span class="o">(</span><span class="k">fun</span> <span class="o">(</span><span class="n">l</span><span class="o">,</span> <span class="n">x</span> <span class="o">:</span> <span class="o">(</span><span class="n">timestamp</span> <span class="o">*</span> <span class="n">tez</span><span class="o">)</span> <span class="kt">list</span> <span class="o">*</span> <span class="o">(</span><span class="n">timestamp</span> <span class="o">*</span> <span class="n">tez</span><span class="o">))</span> <span class="o">-></span> <span class="n">x</span> <span class="o">::</span> <span class="n">l</span><span class="o">)</span>
<span class="n">l</span> <span class="o">(</span><span class="bp">[]</span> <span class="o">:</span> <span class="o">(</span><span class="n">timestamp</span> <span class="o">*</span> <span class="n">tez</span><span class="o">)</span> <span class="kt">list</span><span class="o">)</span>
<span class="k">let</span> <span class="n">get_time</span> <span class="o">(</span><span class="n">n</span> <span class="o">:</span> <span class="kt">int</span><span class="o">)</span> <span class="o">:</span> <span class="n">timestamp</span> <span class="o">=</span> <span class="nn">Current</span><span class="p">.</span><span class="n">time</span>
<span class="k">let</span> <span class="n">prune</span> <span class="o">(</span><span class="n">threshold</span><span class="o">,</span> <span class="n">queue</span> <span class="o">:</span> <span class="n">tez</span> <span class="o">*</span> <span class="n">queue_t</span><span class="o">)</span> <span class="o">:</span> <span class="kt">bool</span> <span class="o">*</span> <span class="n">tez</span> <span class="o">*</span> <span class="n">queue_t</span> <span class="o">=</span>
<span class="k">match</span> <span class="n">queue</span><span class="o">.</span><span class="n">left</span> <span class="k">with</span>
<span class="bp">[]</span> <span class="o">-></span> <span class="k">if</span> <span class="nn">List</span><span class="p">.</span><span class="n">size</span> <span class="n">queue</span><span class="o">.</span><span class="n">right</span> <span class="o">=</span> <span class="mi">0</span><span class="n">n</span>
<span class="k">then</span> <span class="n">stop</span> <span class="o">(</span><span class="n">threshold</span><span class="o">,</span> <span class="o">{</span><span class="n">left</span> <span class="o">=</span> <span class="n">empty_queue_comp</span><span class="o">;</span> <span class="n">right</span> <span class="o">=</span> <span class="n">empty_queue_comp</span><span class="o">})</span>
<span class="k">else</span> <span class="n">continue</span> <span class="o">(</span><span class="n">threshold</span><span class="o">,</span> <span class="o">{</span><span class="n">left</span> <span class="o">=</span> <span class="n">reverse</span> <span class="n">queue</span><span class="o">.</span><span class="n">right</span><span class="o">;</span> <span class="n">right</span> <span class="o">=</span> <span class="n">empty_queue_comp</span><span class="o">})</span>
<span class="o">|</span> <span class="n">hd</span><span class="o">::</span><span class="n">left_tl</span> <span class="o">-></span> <span class="k">let</span> <span class="n">ts</span> <span class="o">=</span> <span class="n">hd</span><span class="o">.</span><span class="mi">0</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">amount</span> <span class="o">=</span> <span class="n">hd</span><span class="o">.</span><span class="mi">1</span> <span class="k">in</span>
<span class="k">if</span> <span class="n">ts</span> <span class="o">></span> <span class="nn">Current</span><span class="p">.</span><span class="n">time</span>
<span class="k">then</span> <span class="n">stop</span> <span class="o">(</span><span class="n">threshold</span><span class="o">,</span> <span class="n">queue</span><span class="o">)</span>
<span class="k">else</span> <span class="n">continue</span> <span class="o">(</span><span class="n">threshold</span> <span class="o">+</span> <span class="n">amount</span><span class="o">,</span> <span class="o">{</span><span class="n">left</span> <span class="o">=</span> <span class="n">left_tl</span><span class="o">;</span> <span class="n">right</span> <span class="o">=</span> <span class="n">queue</span><span class="o">.</span><span class="n">right</span><span class="o">})</span>
<span class="k">let</span> <span class="n">transfer</span> <span class="o">(</span><span class="n">trp</span><span class="o">,</span> <span class="n">sto</span> <span class="o">:</span> <span class="n">transfer_param</span> <span class="o">*</span> <span class="n">storage</span><span class="o">)</span> <span class="o">:</span> <span class="n">operation</span> <span class="kt">list</span> <span class="o">*</span> <span class="n">storage</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">key</span> <span class="o">=</span> <span class="n">trp</span><span class="o">.</span><span class="mi">0</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">sign</span> <span class="o">=</span> <span class="n">trp</span><span class="o">.</span><span class="mi">1</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">transfers</span> <span class="o">=</span> <span class="n">trp</span><span class="o">.</span><span class="mi">2</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">kh</span> <span class="o">=</span> <span class="n">trp</span><span class="o">.</span><span class="mi">3</span> <span class="k">in</span>
<span class="k">if</span> <span class="nn">Crypto</span><span class="p">.</span><span class="n">hash_key</span> <span class="n">key</span> <span class="o">=</span> <span class="n">sto</span><span class="o">.</span><span class="n">key_user</span> <span class="o">&&</span> <span class="nn">Crypto</span><span class="p">.</span><span class="n">check</span> <span class="n">key</span> <span class="n">sign</span> <span class="o">(</span><span class="nn">Bytes</span><span class="p">.</span><span class="n">pack</span> <span class="o">(</span><span class="n">transfers</span><span class="o">,</span> <span class="n">sto</span><span class="o">.</span><span class="n">salt_user</span><span class="o">))</span> <span class="k">then</span>
<span class="k">let</span> <span class="n">current_threshold</span><span class="o">,</span> <span class="n">queue</span> <span class="o">=</span> <span class="nn">Loop</span><span class="p">.</span><span class="n">fold_while</span> <span class="n">prune</span> <span class="o">(</span><span class="n">sto</span><span class="o">.</span><span class="n">current_threshold</span><span class="o">,</span> <span class="n">sto</span><span class="o">.</span><span class="n">queue</span><span class="o">)</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">total_amount</span> <span class="o">=</span> <span class="nn">List</span><span class="p">.</span><span class="n">fold</span> <span class="o">(</span><span class="k">fun</span> <span class="o">(</span><span class="n">acc</span><span class="o">,</span> <span class="n">amount_dest</span> <span class="o">:</span> <span class="n">tez</span> <span class="o">*</span> <span class="o">(</span><span class="n">tez</span> <span class="o">*</span> <span class="kt">unit</span> <span class="n">contract</span><span class="o">))</span> <span class="o">-></span> <span class="n">amount_dest</span><span class="o">.</span><span class="mi">0</span> <span class="o">+</span> <span class="n">acc</span><span class="o">)</span> <span class="n">transfers</span> <span class="mi">0</span><span class="n">mutez</span> <span class="k">in</span>
<span class="k">let</span> <span class="n">ops</span> <span class="o">=</span> <span class="nn">List</span><span class="p">.</span><span class="n">map</span>
<span class="o">(</span><span class="k">fun</span> <span class="o">(</span><span class="n">amount</span><span class="o">,</span> <span class="n">destination</span> <span class="o">:</span> <span class="n">tez</span> <span class="o">*</span> <span class="kt">unit</span> <span class="n">contract</span><span class="o">)</span> <span class="o">-></span>
<span class="nn">Operation</span><span class="p">.</span><span class="n">transaction</span> <span class="kt">unit</span> <span class="n">amount</span> <span class="n">destination</span><span class="o">)</span>
<span class="n">transfers</span> <span class="k">in</span>
<span class="o">(</span><span class="n">ops</span><span class="o">,</span>
<span class="o">{</span> <span class="n">key_user</span> <span class="o">=</span> <span class="n">kh</span><span class="o">;</span>
<span class="n">key_administrator</span> <span class="o">=</span> <span class="n">sto</span><span class="o">.</span><span class="n">key_administrator</span> <span class="o">;</span>
<span class="n">current_threshold</span> <span class="o">=</span> <span class="n">current_threshold</span> <span class="o">-</span> <span class="n">total_amount</span><span class="o">;</span>
<span class="n">time_window</span> <span class="o">=</span> <span class="n">sto</span><span class="o">.</span><span class="n">time_window</span><span class="o">;</span>
<span class="n">queue</span> <span class="o">=</span> <span class="o">{</span><span class="n">left</span> <span class="o">=</span> <span class="n">queue</span><span class="o">.</span><span class="n">left</span><span class="o">;</span> <span class="n">right</span> <span class="o">=</span> <span class="o">(</span><span class="nn">Current</span><span class="p">.</span><span class="n">time</span><span class="o">,</span> <span class="n">total_amount</span><span class="o">)</span> <span class="o">::</span> <span class="n">queue</span><span class="o">.</span><span class="n">right</span><span class="o">};</span>
<span class="n">salt_user</span> <span class="o">=</span> <span class="n">sto</span><span class="o">.</span><span class="n">salt_user</span> <span class="o">+</span> <span class="mi">1</span><span class="n">n</span><span class="o">;</span>
<span class="n">salt_administrator</span> <span class="o">=</span> <span class="n">sto</span><span class="o">.</span><span class="n">salt_administrator</span><span class="o">})</span>
<span class="k">else</span> <span class="o">(</span><span class="n">failwith</span> <span class="s2">"bad_signature"</span> <span class="o">:</span> <span class="n">operation</span> <span class="kt">list</span> <span class="o">*</span> <span class="n">storage</span><span class="o">)</span>
</code></pre></div>
<p>The <code>transfer</code> entrypoint, illustrated by the CameLIGO pseudocode above, receives a public key, a list of new
transfer orders and a signature of this list. The contract verifies
that the hash of this key equals the stored key hash of the user. It
also verifies that the signature of the payload is valid and created
by the user. If either of these conditions are unfulfilled, the
contract fails.</p>
<p>The contract maintains a queue of all previous transfers that fell
within the time window at the previous execution of the contract. In
the <code>transfer</code> entrypoint, this list is traversed, and all members of
this queue that are now outside the time window are removed.</p>
<p>The contract also maintains the <em>current limit</em>, i.e., the amount of
funds that can be spent at this moment. The current limit is now updated
by adding to it the sum of the funds in the removed elements in the
previous step.</p>
<p>If the user attempts to overspend, i.e., if the total of the funds
in the new list of transfers exceeds the current limit, then
the contract will fail at this point. Otherwise, meaning that the list
of ordered transfers are allowed by the time window and spending limit,
then a <code>TRANSFER_TOKEN</code> operation is emitted for each ordered
transfer with the specified amount and destination.</p>
<p><strong>Example</strong>. To illustrate the behavior of the <code>transfer</code> entrypoint,
consider a contract that is configured so that no more than 30 ꜩ can be spent every 40 minutes.
In other words, the time limit is set to 40 minutes and the initial
limit to 30 ꜩ. Assume the user has transferred 3 ꜩ from the
contract at timestamp 12:20 (transfer <code>t1</code>), and 10 ꜩ at timestamp
12:40 (transfer <code>t2</code>).
After performing the last transfer, the current limit is set to 30 - 10 - 3 = 17 ꜩ.
We
can illustrate the state of the contract by a timeline, as below. The timeline is decorated with
all past transfers, and the part of the timeline that falls in the time window is highlighted
in blue.</p>
<p><img alt="Transfer entrypoint, initial state" src="images/slc-verif/ep-transfer-1.png"></p>
<p>Now, say the user attempts to transfer 30 ꜩ at 13:10 (transfer
<code>t3</code>). The contract traverses the queue of past transfers. It finds
that transfer <code>t1</code> falls outside the time window, prunes it, and so
increments the current threshold by 3 from 17 to 20 ꜩ. Nevertheless,
the amount of the transfer (30 ꜩ) exceeds the current limit (20
ꜩ), and hence the request is rejected. The contract execution is
cancelled, its state is unchanged and no operation is emitted.</p>
<p><img alt="Transfer entrypoint, invalid transfer" src="images/slc-verif/ep-transfer-2.png"></p>
<p>The user must opt for a more tempered spending: this time, at
timestamp 13:20, they request a transfer of 15 ꜩ (transfer <code>t4</code>). Again, the history
is traversed and as before the current limit is incremented
to 20 ꜩ. Now, the amount of the transfer 15 ꜩ falls snugly under the
current limit of 20 ꜩ. Consequently the transfer is accepted, a <code>TRANSFER_TOKENS</code> operation is
issued, and the queue and current limit are updated:</p>
<p><img alt="Transfer entrypoint, valid transfer" src="images/slc-verif/ep-transfer-3.png"></p>
<h3 id="receive"><a class="toclink" href="#receive">Receive</a></h3>
<p>The third entrypoint, <code>receive</code> is used to transfer funds to the
contract without provoking any other changes in its state. This
entrypoint emits no operations and does not modify the storage. This
is succinctly expressed in CameLIGO pseudocode:</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="n">receive</span> <span class="o">(</span><span class="n">p</span><span class="o">,</span> <span class="n">s</span> <span class="o">:</span> <span class="kt">unit</span> <span class="o">*</span> <span class="n">storage</span><span class="o">)</span> <span class="o">:</span> <span class="n">operation</span> <span class="kt">list</span> <span class="o">*</span> <span class="n">storage</span> <span class="o">=</span> <span class="o">(</span><span class="bp">[]</span> <span class="o">:</span> <span class="n">operation</span> <span class="kt">list</span><span class="o">),</span> <span class="n">s</span>
</code></pre></div>
<h2 id="approaches-to-formal-verification"><a class="toclink" href="#approaches-to-formal-verification">Approaches to Formal Verification</a></h2>
<p>The criticality of the Cortez wallet, the security motivation of the
spending limit contract, and the difficulties of modifying
already-deployed smart contracts make <span class="caps">SLC</span> a prime target for formal
verification.
In addition,
the optimized representation of the queue induces a gap between the intuitive behavior of the contract and its implementation, which makes verification non-trivial.</p>
<p>While code review and testing are capable of detecting the presence of bugs,
they can never completely guarantee the absence of undesired behaviors.
Making sure programs always behave as expected requires a more rigorous and systematic approach.
This is the mainspring of <strong>formal methods</strong>. At the heart of any formal method is the
specification of a system: the definition of its desired behavior in a
rigorous mathematics-based – formal – flavor.</p>
<p>If a program is seen as a sentence in a programming language, then the
<strong>semantics</strong> of this programming language is the rules that give a
sense to this program, i.e. its behaviors. Hence, verifying a program
is to verify that its semantics complies with its specification. We
now briefly discuss five common formal methods: model checking,
abstract interpretation, mechanized verification,
correctness-by-construction, and deductive verification.</p>
<p><strong>Model checking</strong> is a model-based formal method.
Instead of reasoning on the program, verification is applied to a model:
a transition state machine that enables the inspection of every possible execution path
of the program. The specification, typically written in temporal logics,
can then either been verified on this model or a counter-example is exhibited.
The drawback of model checking is the combinatorial explosion of paths that must be explored,
whose number is proportional to the size of the model.</p>
<p><strong>Abstract interpretation</strong> proposes to reason on an abstract
semantics instead of reasoning on the concrete execution
semantics. The purpose of abstract semantics is to provide dedicated
constructions to the specification to be proven. Consequently,
the abstract interpretation eases the verification. The difficulty here
resides in designing an appropriate and sound approximation, a complex task
that is also very dependent on the properties to prove.</p>
<p><strong>Mechanized verification</strong>. It is possible to use a proof assistant
in order to mechanically assist the construction of a proof
establishing the correctness of a program. In this case, the semantics
of the programming language used, and the specification of the program
is written in the proof assistant’s logic and the proof of correctness
is constructed using the proof assistant’s tactics language.</p>
<p><strong>Correctness-by-construction</strong> ensures compliance
of the program with its specification using a top-down approach.
The implementation is done
in a theorem prover that (i) enables extraction of an executable program from the
specification and (ii) enables the mechanized verification of the program.
The program is then <em>formally certified</em>: it comes with its replayable formal proof.</p>
<p><strong>Deductive verification</strong> enables verifying the compliance of the
program with its specification directly on the behavior of the program
thanks to a Hoare logics<sup id="fnref:4"><a class="footnote-ref" href="#fn:4">4</a></sup> that enables reasoning on pre- and
post-state of a program along with its semantics. The specification of
the program is then expressed through properties that holds on the
pre-state of the program execution, called <em>precondition</em> and
properties that holds on the post-state of its execution, called
<em>postcondition</em>. Together, the precondition, program and
postcondition form a Hoare triplet. While Hoare logics is a
relation that might not terminate, <em>weakest precondition calculus</em>
is a function that computes the weakest precondition for a program and
a postcondition in order to form a valid Hoare triplet. During the
weakest precondition calculus proof obligations are produced. In
mainstream tools, they are dispatched to automatic provers that either
validate them, refute them or cannot decide.
While easing the proof engineer’s work, using automatic provers presents two limits when used in
conjunction with interactive proving.
First, automatic provers typically do not adhere to a common
proof format. Consequently, the proof assistants must implement
a wide variety of formats, or trust the automatic prover blindly.
Second, when the automatic prover can’t decide,
the proof engineer has to
provide a counter-example or
handle the proof manually in a proof assistant. In that case, the specification that is originally designed to suit an automatic prover
may be inconvenient for the proof engineer working in a proof assistant.</p>
<p>We have formally specified the Spending Limit Contract,
in other words, described precisely and unambiguously its desired
behavior. Using the weakest precondition calculus of
<a href="https://gitlab.com/nomadic-labs/mi-cho-coq/">Mi-Cho-Coq</a>, we
have then proved that the <span class="caps">SLC</span> contract fulfills this
specification. Not only do we ensure that <span class="caps">SLC</span> “does what it is
supposed to do”, that is, throttle spending. We also ensure that
its execution does not cause any unintended run-time errors<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>.</p>
<h2 id="mi-cho-coq"><a class="toclink" href="#mi-cho-coq">Mi-Cho-Coq</a></h2>
<p align="center">
<img src="images/slc-verif/mi-cho-coq.png" />
</p>
<p><a href="https://gitlab.com/nomadic-labs/mi-cho-coq/">Mi-Cho-Coq</a> is a
formalization of the semantics of the Tezos smart contract language
<a href="http://tezos.gitlab.io/whitedoc/michelson.html">Michelson</a> in Coq.
Michelson is a strongly typed, pure functional and stack-based programming language.
Each Michelson instruction can be seen as a function from an input stack of working data to an output stack, in a blockchain environment. The blockchain environment gives current state of the blockchain, containing information such as the current timestamp and the balance of the executing contract.</p>
<p>Whereas Michelson has always had a
<a href="http://tezos.gitlab.io/whitedoc/michelson.html#language-semantics">pen-and-paper semantics</a>,
encoding it in a proof assistant such as Coq helps to
ensure that all corner cases are unambiguously specified. Mi-Cho-Coq also enables deductive verification of Michelson programs thanks to
its weakest precondition calculus
that enables using the high-level expressiveness
of Coq. This requires that all proof obligations are discharged by the proof engineer in Coq.</p>
<p>In Mi-Cho-Coq, the semantics of Michelson programs is given by an
evaluation function <code>eval</code> that is similar to the interpreter
embedded in the <code>tezos-node</code>:</p>
<div class="highlight"><pre><span></span><code> <span class="kn">Fixpoint</span> <span class="n">eval</span> <span class="c">(* omitted parameters *)</span>
<span class="o">(</span><span class="n">env</span> <span class="o">:</span> <span class="o">@</span><span class="n">proto_env</span> <span class="n">param_ty</span><span class="o">)</span>
<span class="o">(</span><span class="n">i</span> <span class="o">:</span> <span class="n">instruction</span> <span class="n">param_ty</span> <span class="n">tff0</span> <span class="n">A</span> <span class="n">B</span><span class="o">)</span>
<span class="o">(</span><span class="n">fuel</span> <span class="o">:</span> <span class="n">Datatypes.nat</span><span class="o">)</span>
<span class="o">(</span><span class="n">SA</span> <span class="o">:</span> <span class="n">stack</span> <span class="n">A</span><span class="o">)</span>
<span class="o">{</span><span class="k">struct</span> <span class="n">fuel</span><span class="o">}</span> <span class="o">:</span> <span class="n">M</span> <span class="o">(</span><span class="n">stack</span> <span class="n">B</span><span class="o">)</span> <span class="o">:=</span>
<span class="k">match</span> <span class="n">fuel</span> <span class="k">with</span>
<span class="o">|</span> <span class="n">O</span> <span class="o">=></span> <span class="n">Failed</span> <span class="o">_</span> <span class="n">Out_of_fuel</span>
<span class="o">|</span> <span class="n">S</span> <span class="n">n</span> <span class="o">=></span>
<span class="k">match</span> <span class="n">i</span><span class="o">,</span> <span class="n">SA</span><span class="o">,</span> <span class="n">env</span> <span class="k">with</span>
<span class="c">(* ... *)</span>
<span class="c">(* Semantics of the NOOP instruction *)</span>
<span class="o">|</span> <span class="n">NOOP</span><span class="o">,</span> <span class="n">SA</span><span class="o">,</span> <span class="o">_</span> <span class="o">=></span> <span class="n">Return</span> <span class="n">SA</span>
<span class="c">(* Semantics of the SEQ instruction *)</span>
<span class="o">|</span> <span class="n">SEQ</span> <span class="n">i1</span> <span class="n">i2</span><span class="o">,</span> <span class="n">SA</span><span class="o">,</span> <span class="n">env</span> <span class="o">=></span>
<span class="k">let</span><span class="o">!</span> <span class="n">r</span> <span class="o">:=</span> <span class="n">eval</span> <span class="n">env</span> <span class="n">i1</span> <span class="n">n</span> <span class="n">SA</span> <span class="k">in</span>
<span class="n">eval</span> <span class="n">env</span> <span class="n">i2</span> <span class="n">n</span> <span class="n">r</span>
<span class="c">(* ... *)</span>
</code></pre></div>
<p>The semantics of a Michelson instruction <code>i</code> is given by the recursive
function <code>eval</code>. This is a function that takes as input a blockchain state <code>env</code>, an input
stack <code>SA</code> and a <code>fuel</code> argument, and returns an output stack.
To maintain soundness of proofs expressed in Coq, all functions must be total and terminating.
To fulfill the first requirement, the result of the <code>eval</code> is wrapped in the error monad: <code>M (stack B)</code>.
This simply means that <code>eval</code> either returns an output stack of type <code>stack B</code> or
fails, which happens in the case of run-time errors or lack of <code>fuel</code>.
The <code>fuel</code> (not to be confused with gas) argument ensures that the <code>eval</code> function is terminating, by
bounding the maximum number of recursive steps that the function is allowed to take.</p>
<p>The snippet above demonstrates the semantics of the instruction
<code>NOOP</code>, that simply returns the input stack <code>SA</code> unchanged. The
semantics of the sequencing instruction <code>SEQ i1 i2</code> consists of first
executing <code>i1</code>. If successful, a new stack <code>r</code> is obtained on which
<code>i2</code> is executed.</p>
<h2 id="verifying-spending-limit-in-mi-cho-coq"><a class="toclink" href="#verifying-spending-limit-in-mi-cho-coq">Verifying Spending Limit in Mi-Cho-Coq</a></h2>
<p>As the interpreter of Mi-Cho-Coq looks similar to the one already implemented in
the <code>tezos-node</code>, one might wonder why go through greats lengths to
rewrite the same thing? The answer is that thanks to the rich
type-system of Coq, we can now state and prove properties on the
behavior of <code>eval</code>. For instance, we can formalize the <em>informal</em> specification of
Spending Limit Contract sketched out above in Coq, and then assert that each execution
of the contract should adhere to the specification. This assertion translates into the
following theorem, as expressed in Coq:</p>
<div class="highlight"><pre><span></span><code><span class="kn">Theorem</span> <span class="n">slc_correct</span> <span class="n">input</span> <span class="n">output</span> <span class="o">:</span>
<span class="k">forall</span> <span class="n">fuel</span><span class="o">,</span>
<span class="n">fuel</span> <span class="o">>=</span> <span class="n">slc_fuel_bound</span> <span class="n">input</span> <span class="o">-></span>
<span class="n">eval</span> <span class="n">env</span> <span class="n">slc</span> <span class="n">fuel</span> <span class="n">input</span> <span class="o">=</span> <span class="n">Return</span> <span class="n">output</span> <span class="o"><-></span>
<span class="n">slc_spec</span> <span class="n">fuel</span> <span class="n">input</span> <span class="n">output</span><span class="o">.</span>
</code></pre></div>
<p>We let <code>slc_fuel</code> be a function giving a bound on the number of steps
necessary to execute the contract, that depends notably on the length of
the queue of past transfers. The script of the <span class="caps">SLC</span> smart contract is <code>slc</code> and
its specification is given by <code>slc_spec</code>. This theorem states that
when given a sufficient amount of fuel, the spending limit
contract when executed on the <code>input</code> stack will return the stack <code>output</code>
if, and only if (denoted <code><-></code> in Coq), this input and this output are related by the <code>slc_spec</code> relation.</p>
<p>The semantics of a Michelson contract is a function from an input (pre-) stack to an output (post-)stack,
and the contract is a sequence of instructions.
Hence, it is adequate to use the weakest precondition calculus of Mi-Cho-Coq.
This calculus translates the statement <code>eval env slc fuel input = Return output</code>
into a logical relation between the input and output stacks.
For the details on how we translate the informal specification given
above, into its formalized version <code>slc_spec</code> as well as the actual
proof of the theorem using the weakest precondition calculus, we refer to the ongoing
<a href="https://gitlab.com/nomadic-labs/mi-cho-coq/merge_requests/51">Mi-Cho-Coq merge request</a>
containing the full development.</p>
<p align="center" style="margin: 40px 20px">
<a href="images/slc-verif/coq-slc_correct.gif">
<img style="margin-right: 60px; border: 2px solid #eee"
src="images/slc-verif/coq-slc_correct.gif" /></a>
</p>
<p>To summarize, we have elaborated an informal specification of the
Spending Limit Contract, that we have then formalized in Coq. We have then
verified that the Spending Limit Contract conforms to this specification using Mi-Cho-Coq.
In addition to showing that <span class="caps">SLC</span> behaves as expected, this also proves that the Spending Limit Contract
cannot provoke any unforeseen run-time errors. All-in-all, this work
required roughly three weeks of work, and the full development consists of
1377 lines of Coq code. Of these lines, 269 consist of auxiliary
lemmas that we hope will be helpful for proving future contracts.
Additionally, 518 lines concern the specification and verification of
the code manipulating the queue of past transfers. As this data
structure, and its optimized representation used in <span class="caps">SLC</span>, is standard, we
can reasonably assume the reusability of these proofs.</p>
<h1 id="next-steps"><a class="toclink" href="#next-steps">Next Steps</a></h1>
<p>While this development allows us to increase trust in the
Spending Limit Contract, and by extension, the Cortez wallet, we have
still only solved one half of the verification puzzle. The
specification we have proved only concerns the <em>functional</em>
correctness of <span class="caps">SLC</span>, that is, its input-output behavior.
However, there is another set of properties
called temporal correctness that concerns the behavior
of a program over a sequence of calls.</p>
<p>For instance, in the case of the Spending Limit Contract, we would
like to be able to specify and prove that the funds in the contract
can <em>eventually</em> be used. This type of property cannot be assured by
simply establishing a relation between the contract’s input and its
computed output. Instead, the behavior of the contract must be
investigated over all possible sequences of calls. This kind of
reasoning is typically the domain of tools called
<a href="https://en.wikipedia.org/wiki/Model_checking">model checkers</a>. Case
in point, the temporal properties of an auction contract for Tezos
have been verified<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup> using the <a href="http://cubicle.lri.fr/">Cubicle</a> model checker. However, temporal properties can
also be expressed and proved deductively in Coq, as have been
done for <a href="https://arxiv.org/abs/1801.00687">Scilla</a>, another smart-contract
language. The next step in our plan to assure the trust in
Cortez is to complete the verification puzzle for <span class="caps">SLC</span> by specifying
and proving its desired temporal properties, either through model
checking or by deductive proof.</p>
<p>Another important missing link is that between these formal proofs,
currently stored in a <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/blob/master/src/contracts_coq/multisig.v">GitLab
repository</a>,
and the actual smart contracts that live <a href="https://better-call.dev/main/KT1DUfaMfTRZZkvZAYQT5b3byXnvqoAykc43/script">on the block
chain</a>.
Consider a user that <a href="https://tezos.stackexchange.com/questions/1696/how-to-verify-the-manager-tz-script-code">wants to ensure the
validity of a purported correctness proof of a deployed contract</a>.
They must first install Coq and Mi-Cho-Coq and have Coq verify the
correctness proof of the contract. Then, they must download the
contract and verify that it is indeed exactly the same as the one
proved in the Mi-Cho-Coq distribution.</p>
<p>We envision a smart contract explorer that automatically associates
contract hashes with properties, such as correctness specifications
and their associated proofs. Users can add new proofs and the explorer
verifies their validity. Additionally, the explorer provides
instructions allowing other users to replicate the proofs. This kind of
explorer would be important to illustrate the nature and importance
of formal methods to end-users, and would increase overall confidence
in smart-contract based systems.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>A run-time error may result from a programming error but can
also be caused intentionally to halt an execution, for instance when
erroneous parameters have been supplied. This is used in <span class="caps">SLC</span> in the
transfer entrypoint. When the user requests a transfer, then its sum is
subtracted from current threshold. Unless the threshold is superior or
equal to the sum, this subtraction results in a tez overflow that
halts execution. This implementation effectively disallows overspending. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>Sylvain Conchon, Alexandrina Korneva and Fatiha Zaidi. Verifying
Smart Contracts with Cubicle. Presented at
<a href="https://sites.google.com/view/fmbc/home"><span class="caps">FMBC</span>‘2019</a>, publication
forthcoming. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>Chris Okasaki, Purely Functional Data Structures, p. 42. <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:4">
<p><span class="caps">C. A. R.</span> Hoare. 1969. An axiomatic basis for computer programming. Commun. <span class="caps">ACM</span> 12, 10 (October 1969), 576-580. <span class="caps">DOI</span>=http://dx.doi.org/10.1145/363235.363259 <a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
</ol>
</div>Meanwhile at Nomadic Labs #52020-01-03T15:00:00+01:002020-01-03T15:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2020-01-03:/meanwhile-at-nomadic-labs-5.html<p>Summary:</p>
<p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>In this fifth series of ‘’Meanwhile at Nomadic Labs’’, we discuss recent achievements including some projects we have been working on.</p>
<p>December was a demanding month with a lot of ongoing development. Like we briefly mentioned last month, Arvid Jakobsson and Zaynah Dargaye worked on formally verifying the spending …</p><p>Summary:</p>
<p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>In this fifth series of ‘’Meanwhile at Nomadic Labs’’, we discuss recent achievements including some projects we have been working on.</p>
<p>December was a demanding month with a lot of ongoing development. Like we briefly mentioned last month, Arvid Jakobsson and Zaynah Dargaye worked on formally verifying the spending limit contract within the Cortez Wallet. We are proud announce that the contract has been formally verified and an updated version of Cortez has gone live on the android play store. This means you can already set spending limits in your Tezos wallet on Android (provided you are using Cortez). Stay tuned for a detailed blog post and an updated <span class="caps">IOS</span> version soon.</p>
<p>Mehdi Bouaziz, Alexandre Doussot and Hadrien Zerah, two of our research engineers and our adoption manager, attended <a href="http://www.riadi.rnu.tn/BlockChain19/index.html">L’école d’automne Blockchain’19</a>, a 3-day blockchain course in Tunisia. During this event, Mehdi and Alexandre organized a Tezos workshop while Hadrien took part in the ‘’enterprise panel’’.</p>
<p>We sponsored a Tezos meetup in Barcelona on December 5th. Pietro Abate gave an <a href="https://www.youtube.com/watch?v=mU1V2kPhO50">introduction to Tezos</a> and Marco Stronati presented some thoughts on <a href="https://www.youtube.com/watch?v=8KCr6Z7qIV8">Babylon and the way forward to Carthage</a>.</p>
<p>Together with Cryptium Labs, we released a new and improved version of the Carthage protocol, which was successfully injected on December 11th. Additionally we launched Carthagenet, a dedicated test network, which passed quorum and is now up and running since December 12th. As usual, we encourage more people to join Carthagenet and come test with us! </p>
<p>The mainnet branch was updated to be in sync with the mainnet staging branch, offering a significant reduction of the size of the context, going from more than 220G to around 40G for <strong>archive nodes</strong>. More information can be found in our <a href="https://forum.tezosagora.org/t/the-mainnet-branch-has-been-updated-and-synced-with-mainnet-staging/1498">Agora post</a>.
The storage now uses Irmin2 together with Irmin-pack. These open source tools are developed by Tarides within the <a href="https://mirage.io/">MirageOS project</a>.</p>
<p>Guillaume Claret presented Coq of OCaml at the Jussieu campus, during the latest OCaml <a href="https://discuss.ocaml.org/t/next-oups-meetup-december-18th-2019-in-paris/4850"><span class="caps">OUPS</span> meetup</a> on December 18th. Nomadic Labs sponsored the event.</p>
<p>On December 16th, we published <a href="https://research-development.nomadic-labs.com/a-new-reward-formula-for-carthage.html">A new reward formula for Carthage</a>, an in-depth review of the new proposed reward formula for Carthage. Shortly after, we published <a href="https://research-development.nomadic-labs.com/sapling-integration-in-tezos-tech-preview.html">Sapling integration in Tezos - Tech Preview</a> which goes into more detail about Sapling, the importance of privacy preserving transactions and our plan to integrate this technology into Tezos on a smart contract basis.</p>
<p>2019 has been an awesome year for Nomadic Labs, Tezos and the entire ecosystem. For a recap you can have a look at the <a href="https://twitter.com/hashtag/Tezos2019?src=hashtag_click">#Tezos2019 campaign</a> on twitter. You can also check directly on our <a href="https://twitter.com/LabosNomades">twitter profile</a> for a recap of the most important Nomadic Labs related events in 2019. Simply keep an eye out for the <strong>#Tezos2019</strong> hashtag.
We wish everybody a Happy New Year.</p>Sapling integration in Tezos - Tech Preview2019-12-24T18:00:00+01:002019-12-24T18:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2019-12-24:/sapling-integration-in-tezos-tech-preview.html<p>We are happy to announce a first technology preview of our integration
in Tezos of the core of the Sapling protocol developed by the ZCash project.
By extending the Michelson smart contract language, this work allows
for the exchange of digital assets in a privacy preserving way.</p>
<h1 id="why-sapling"><a class="toclink" href="#why-sapling">Why Sapling?</a></h1>
<p>In recent years, we’ve seen much progress towards enabling privacy-preserving
payments on public ledgers, both in academic research and in the real world
deployement with projects such as <a href="https://z.cash">Zcash</a>,
<a href="https://www.getmonero.org/">Monero</a>, or <a href="https://www.aztecprotocol.com/">Aztec</a>.</p>
<p>In our opinion the <a href="https://z.cash/">Zcash project</a> and in particular the Sapling protocol
developed by the <a href="https://electriccoin.co/">Electric Coin Company</a> represents the state
of the art to enable privacy-preserving transactions that’s available, in production, today.</p>
<p>As a result we have been working to integrate the Sapling
protocol into Tezos as an instruction into the Michelson smart contract
language for the past year.</p>
<h1 id="what-is-sapling"><a class="toclink" href="#what-is-sapling">What is Sapling?</a></h1>
<p>Sapling is a <a href="https://github.com/zcash/zips/blob/master/protocol/sapling.pdf">protocol</a>
enabling privacy preserving transactions of fungible tokens in a decentralised
environment. It was designed and implemented by the Electric Coin
Company as the last iteration over a series of previous protocols and
academic works starting with the <a href="http://zerocoin.org/media/pdf/ZerocoinOakland.pdf">Zerocoin seminal
paper</a>.
Thanks to significant performance improvements of Sapling, privacy preserving
transactions are finally practical, with approximately ~0.5s needed to
craft a transaction and ~10 ms to verify it on a regular laptop.
Note that the verification time is particularly important in a blockchain environment.</p>
<h2 id="the-keys"><a class="toclink" href="#the-keys">The keys</a></h2>
<p>Sapling provides a slightly different set of keys from the ones
existing today in Tezos. Each user generates a <em>spending key</em> which allows to perform payments
(the equivalent of a <em>secret key</em> in Tezos) and a <em>viewing key</em> which
allows to see the payments received or emitted.
The <em>viewing key</em> can for example be shared with an auditor for
regulatory compliance purposes without having to give the <em>spending key</em>.
In order to receive a payment a user can generate an <em>address</em> from
their viewing key and communicate it to the payer, much like a tz
address today.
Sapling addresses are diversified, meaning that from the same viewing key
it is possible to derive new addresses which are not correlated
to the previous one.
This avoids the inherent leak of privacy happening with one public key where your
different payers collude to realise that they are paying the same person.</p>
<h2 id="shielded-transactions"><a class="toclink" href="#shielded-transactions">Shielded transactions</a></h2>
<p>Once our keys are set up we can start sending and receiving privacy
preserving transactions, which are usually referred to as <em>shielded</em>.
In a shielded transaction the amount transferred,
the sender and the recipient are hidden. Only the sender and the
receiver, using their viewing keys are aware of the details of the
transaction.
The state of the ledger for shielded transactions is usually referred
to as the <em>shielded pool</em>.</p>
<p>In order to bridge a transparent ledger, like Tezos today, with a
shielded pool, we need to shield and unshield transparent tokens. While
shielded transactions are privacy preserving as explained
above, any operation between the transparent ledger and the shielded
pool does reveal that a certain tz address has just un/shielded a
certain amount of tez.</p>
<p>It should be noted that as for every Privacy Enhancing Technology,
there is no perfect system.
It is important to follow good practices and to have a good
understanding of what information your system is leaking.
We will expand on this later.</p>
<h1 id="sapling-in-tezos"><a class="toclink" href="#sapling-in-tezos">Sapling in Tezos</a></h1>
<p>What makes our integration in Tezos special is the fact that
Sapling is exposed in the smart contract language Michelson.
This makes it possible to effectively have a shielded pool in each
contract, where tokens can be created and destroyed programmatically.
A use case is for privacy-preserving STOs where any logic
can be applied for the creation and destruction of tokens and where
users can exchange their tokens while maintaining their privacy.</p>
<h2 id="michelson"><a class="toclink" href="#michelson">Michelson</a></h2>
<p>We created a very high level Michelson instruction called
<code>sapling_verify_update</code>, and two Michelson types called <code>sapling_state</code>
and <code>sapling_transaction</code>. The instruction takes a state (i.e. a
shielded pool) and a transaction, checks that the transaction is valid
with regard to the current state, and returns an updated state.
It also returns the <em>balance</em> of the transaction, which
is the difference between the number of shielded tokens created and
destroyed. If the latter is bigger than the former the balance is
negative and vice versa. This allows to program the logic for the
conversion from and to the shielded pool directly in the smart
contract.
Note that transferring within the shielded pool has a balance of zero
and nothing is revealed of the amount transferred.</p>
<h2 id="example-contract"><a class="toclink" href="#example-contract">Example contract</a></h2>
<p>A simple contract involving shielded transactions is one that accepts
the creation of a shielded token in exchange for <em>n</em> tez.
In this case a user with transparent address <code>tz1-alice</code> will generate
a sapling address <code>sapling-alice</code> and a transaction that transfers <em>n</em> tez
to the contract and asks for a shielded token to be credited to <code>sapling-alice</code>.
After the transaction is processed, the contract is credited n tez,
Alice’s account <code>tz1-alice</code> is debited n tez (plus the usual fees) and
the contract will create a shielded token in its shielded pool.</p>
<p>Once Alice has a shielded token in the pool, she can send any number of
shielded transactions to another shielded address.
In this case, the only visible thing from the exterior is that a tz
address (not necessarily Alice’s) is performing shielded transactions.</p>
<p>If Alice decides to unshield her token, she can send a
transaction that destroys it and the contract will simply send to
<code>tz1-alice</code> <em>n</em> tez from its balance and burn the shielded token from the pool.</p>
<p>As for any smart contract, this template contract can be extended to
adjust its rules programmaticall, interact with other contracts or
with an oracle, etc.</p>
<p>Indeed with this technology it is possible to implement a privacy
preserving version of <a href="https://assets.tqtezos.com/"><span class="caps">FA</span> 1.2</a> and the
possibilities are endless.</p>
<h1 id="privacy-considerations"><a class="toclink" href="#privacy-considerations">Privacy considerations</a></h1>
<h2 id="fees-issue"><a class="toclink" href="#fees-issue">Fees issue</a></h2>
<p>When interacting with any smart contract it’s necessary to pay fees
using a regular tz address and that’s also the case when sending a
shielded transaction.</p>
<p>It is important that once Alice shields her token, she doesn’t use her
<code>tz1-alice</code> to submit her shielded transactions to the contract.
This can be mitigated by using a service that acts as a proxy by
forwarding Alice’s transactions to the contract and paying its fees.
Alice can include in the transaction a shielded amount for the service that
covers the fees plus a small bonus to pay the service.</p>
<h2 id="good-practices"><a class="toclink" href="#good-practices">Good practices</a></h2>
<p>When blending in a group of people, one should always pay attention to the
size and the variety of the group. The size is important because if
Alice and Bob are the only users in the shielded pool, and it only
contains 10 tokens, whenever we see a shielded transaction we can
still infer that Bob and Alice transferred less than 10 tokens.
The variety of a set is important because if a shielded pool is used
exclusively by left-handed people for example, then the mere fact of
interacting with it may reveal your handedness.</p>
<p>We recommend two good practices. First, do not originate a second
contract if another one has the same functionalities, it will split
the anonymity set.</p>
<p>Second, remember that shielding and unshielding are public operations.
A typical anti-pattern is to shield from tz1-alice 15.3 tez, and then
unshield 15.3 tez to tz1-bob. It’s fairly clear from timing and
amounts that Alice transferred 15.3 tez to Bob.</p>
<p>There are a number of more sophisticated techniques to deanonymise
users using timing of operations, network monitoring, side-channels on
clients and analysis of number of inputs/outputs just to mention a few.
We advise users to be familiar with the use of the <span class="caps">TOR</span> network and to
use clients developed specifically to protect their privacy.</p>
<h1 id="status"><a class="toclink" href="#status">Status</a></h1>
<h2 id="what-is-ready"><a class="toclink" href="#what-is-ready">What is ready</a></h2>
<p>We invite more advanced users to try out our
working branch
<a href="https://gitlab.com/metastatedev/tezos/commits/sapling-integration/">sapling-integration</a>.
All the operations can be tested in sandbox using the simple contract
we provide and our integration in the tezos-client.
More detailed instructions can be found in the <a href="https://gitlab.com/metastatedev/tezos/blob/sapling-integration/docs/developer/sapling.rst">documentation</a>.</p>
<h2 id="what-is-almost-ready"><a class="toclink" href="#what-is-almost-ready">What is almost ready</a></h2>
<h3 id="storage"><a class="toclink" href="#storage">Storage</a></h3>
<p>The storage of a smart contract using Sapling effectively stores a
whole “blockchain”, with all the history of transactions. Therefore it
could potentially be large and unwieldy. For this reason we are working on an
efficient implementation much like the one used for big maps.</p>
<h3 id="rpcs"><a class="toclink" href="#rpcs">RPCs</a></h3>
<p>Specific RPCs will be added to interact with the new sapling storage.</p>
<h3 id="gas"><a class="toclink" href="#gas">Gas</a></h3>
<p>Once the storage is ready, it will be possible to run accurate gas
benching and compute the cost of sapling operations.</p>
<h1 id="conclusion-and-acknowledgments"><a class="toclink" href="#conclusion-and-acknowledgments">Conclusion and acknowledgments</a></h1>
<p>We hope that the freedom given by creating a Michelson instruction
will encourage developers to make creative use cases and that this
will only be the start of financial privacy on Tezos.</p>
<p>We wish to particularly thank two groups of people. First, the
cryptography research community that has been working on privacy and
zero-knowledge protocols since the 80’s.
Next, we would like to thanks the Z-Cash project and the Electronic Coin
Company for their great protocol and for taking the time to write a
superb specification (which can be hard to come by in our quickly
moving environment).</p>A new reward formula for Carthage2019-12-16T09:00:00+01:002019-12-16T09:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2019-12-16:/a-new-reward-formula-for-carthage.html<p>Summary:</p>
<p><strong>Note</strong>: <em>This analysis was done with the help of Arthur Breitman and Bruno Blanchet (Inria).
The code used for the analysis can be found at <a href="https://gitlab.com/nomadic-labs/emmy-simulator">this url</a>.</em></p>
<hr>
<h1 id="a-new-reward-formula-for-carthage"><a class="toclink" href="#a-new-reward-formula-for-carthage">A new reward formula for Carthage</a></h1>
<p>In this article, we present a new reward formula that we propose for inclusion in Carthage …</p><p>Summary:</p>
<p><strong>Note</strong>: <em>This analysis was done with the help of Arthur Breitman and Bruno Blanchet (Inria).
The code used for the analysis can be found at <a href="https://gitlab.com/nomadic-labs/emmy-simulator">this url</a>.</em></p>
<hr>
<h1 id="a-new-reward-formula-for-carthage"><a class="toclink" href="#a-new-reward-formula-for-carthage">A new reward formula for Carthage</a></h1>
<p>In this article, we present a new reward formula that we propose for inclusion in Carthage. This new formula is designed to make the network more robust to non-cooperative baking strategies. It does so without changing the total amount of rewards earned by bakers per period of time, i.e. this update should be transparent to bakers and stakers alike.</p>
<p>This work is a follow-up to our previous <a href="https://research-development.nomadic-labs.com/analysis-of-emmy.html">blog post on Emmy+</a>. It takes into account new criteria to evaluate the performance of non-cooperative baking strategies for the dishonest baker, and a new non-cooperative baking strategy called <em>deflationary baking</em> contributed by a researcher who desires to remain anonymous.</p>
<p>We found this new function more resilient to all non-cooperative baking strategies that we currently know. As a side effect, it slightly increases the variance of rewards. The formula is designed to be less punishing to endorsers whose endorsements are included in blocks of non-zero priority.</p>
<p>The plan of this post is as follows.</p>
<ul>
<li>We first give a bit of context explaining how protocol
design relies on assumptions on the <a href="#sec:altpro">notions of profitability</a> held
by bakers.</li>
<li>A particular family of <a href="#sec:noncoop">noncooperative baking strategies</a> is introduced,
corresponding to non-standard notions of profitability. These are the strategies against
which we wish to protect the protocol by picking a new reward function.</li>
<li>We then describe the <a href="#sec:methodology">methodology and rationale</a> that guided us in choosing this new function.</li>
<li>The <a href="#sec:comparison">results of our analysis</a> confirm the good properties of the new reward
function in adversarial settings.</li>
<li>We conclude this post by listing <a href="#sec:RPC">changes to the <span class="caps">RPC</span> <span class="caps">API</span></a> specific to the this new
function that will be of interest to the Tezos developer community.</li>
</ul>
<h2 id="alternative-notions-of-profitability-induce-different-baking-strategies"><a class="toclink" href="#alternative-notions-of-profitability-induce-different-baking-strategies"><a name="sec:altpro"></a> Alternative notions of profitability induce different baking strategies</a></h2>
<p>Baking can be framed as a game in which bakers follow strategies aiming at maximizing their profits, for some notion of profitability. A strategy is called <em>cooperative</em> when a baker bakes publicly, on time and includes all available endorsements. Cooperative baking strategies are the ones that provide the chain with the best security, fastest probabilistic finality, and throughput. Nomadic Labs’ baker (which is currently the most widely used) implements such a strategy.</p>
<p>Emmy+ was engineered relying on the assumption that a baker’s notion of profitability is to maximize their reward, so that the dominant strategy for such bakers is cooperative.</p>
<p>However, there are other notions of profitability that we didn’t consider when designing Emmy+ and which induce bakers preferring these notions to deviate from cooperativeness.
These other strategies are bad for the protocol (eg they slow down the chain) but are more profitable to bakers pursuing this alternative notion of profitability.</p>
<p>Our new reward function is designed to protect against non-cooperative strategies arising from these new notions of profitability.</p>
<p>In the rest of this post, we will denote Emmy+B and Emmy+C the versions of Emmy+ using respectively the Babylon reward function and the new one we’re proposing for Carthage.</p>
<h3 id="notions-of-profitability"><a class="toclink" href="#notions-of-profitability">Notions of profitability</a></h3>
<p>Let’s introduce some notations. We will consider the rewards earned by a non-cooperative baker <span class="math">\(A\)</span> and those earned by the rest of the delegators aggregated under the name <span class="math">\(B\)</span>, over a fixed period of (physical) time. We will denote these respectively <span class="math">\(r_A, r_B\)</span>. We will denote their share of the active stake respectively <span class="math">\(s_A,s_B\)</span>, with <span class="math">\(s_A + s_B = 1\)</span> (hence we make the choice to ignore undelegated tokens in the inflation computation below). The total supply at the beginning of the time period will be denoted <span class="math">\(S\)</span>. For the rest of the article, we take <span class="math">\(S = 800\times10^6\)</span>.</p>
<p>When designing the new function, we considered the following notions of profitability:</p>
<ul>
<li><strong>Absolute rewards (<span class="caps">AR</span>)</strong> is the criteria we used in our previous analysis, which corresponds to the total rewards earned per unit of time. This corresponds to maximizing <span class="math">\(AR = r_A\)</span>.</li>
<li><strong>Inflation adjusted rewards (<span class="caps">IAR</span>)</strong>, also known as the <em>real rate of return</em>. To obtain this function, one considers the rewards relative to the stake, defined as <span class="math">\(rr_A = \frac{r_A}{S \times s_A}\)</span>, and correct them by the inflation over the considered period, defined by <span class="math">\(I =\frac{r_A + r_B}{S}\)</span>. <span class="math">\(IAR\)</span> are defined as relative rewards properly diluted by the inflation, corresponding to the function <div class="math">$$IAR = \frac{rr_A + 1}{I + 1} - 1 = \frac{r_A - s_A(r_A+r_B)}{s_A\times(S+r_a+r_b)}$$</div>
</li>
<li><strong>Rewards share difference (<span class="caps">RSD</span>)</strong> measures the difference in rewards share between the non-cooperative baker and the rest of the network. We define this function as <div class="math">$$RSD = \frac{r_A}{s_A} - \frac{r_B}{s_B} = \frac{r_A}{s_A} - \frac{r_B}{1 - s_A}$$</div> In other words, a non-cooperative baker wanting to maximize this criterion is willing to lose any token, so long as the others lose more in proportion of their stake.</li>
</ul>
<h2 id="non-cooperative-baking-strategies"><a class="toclink" href="#non-cooperative-baking-strategies"><a name="sec:noncoop"></a> Non-cooperative baking strategies</a></h2>
<p>As explained in the previous paragraph, bakers have preferences that manifest concretely as notions of profitability. In a second step, these notions, together with contextual information on the preferences of the other bakers, induce baking strategies. In the rest of this post, we will posit a non-cooperative baker acting in a context where the rest of the stake is entirely held by bakers using a cooperative strategy. The strategies we depict are not necessarily optimal in any sense: they must be understood as lower bounds on how toxic non-cooperative strategies can be. These are the strategies we implemented up to now, and against which we aim to protect the protocol.</p>
<ul>
<li>
<p><strong><span class="caps">AR</span> block stealing</strong> (<code>ar-attacker</code>)
Block stealing is an <span class="caps">AR</span>-oriented non-cooperative baking strategy, in which a baker of priority <span class="math">\(p > 0\)</span> will withhold his endorsements in order to slow down the blocks of priority <span class="math">\(0,..,p-1\)</span> enough for his block to be the fastest one. The decision to perform a steal for an <span class="caps">AR</span>-oriented baker will be predicated on checking that the move is both <em>feasible</em> (in terms of delays) and <em>profitable</em> (in the <span class="caps">AR</span> sense). Emmy+B was specifically designed to make these steals hard to profit from (see <a href="https://research-development.nomadic-labs.com/analysis-of-emmy.html">Emmy+ blog post</a> and a <a href="https://arxiv.org/pdf/1912.02954.pdf">recent paper</a> which also analyzes selfish baking in Emmy+B and obtains similar results).</p>
</li>
<li>
<p><strong>Deflationary baking + <span class="caps">IAR</span> block stealing</strong> (<code>iar-attacker</code>)
The <span class="caps">IAR</span>-oriented non-cooperative strategy we considered consists in combining block stealing together with <em>deflationary baking</em>. Deflationary baking corresponds to actively refusing to include endorsements of other bakers in one’s blocks. A naive deflationary strategy for baker <span class="math">\(A\)</span> is to include his own endorsements plus the minimal amount of endorsements of baker <span class="math">\(B\)</span> so that the incurred delay still ensures that his block is faster than the block of baker <span class="math">\(B\)</span> with highest priority. An <span class="math">\(IAR\)</span> non-cooperative baker will refine this strategy by selecting the <span class="math">\(IAR\)</span>-optimal number of endorsements in his deflationary blocks, and also by stealing blocks when feasible (again, optimizing the number of endorsements for the <span class="math">\(IAR\)</span> measure).</p>
</li>
<li>
<p><strong>Deflationary baking + <span class="caps">RSD</span> block stealing</strong> (<code>rsd-attacker</code>)
Lastly, we considered an <span class="caps">RSD</span>-oriented strategy based on deflationary baking plus <span class="caps">RSD</span>-specific block stealing. Similarly to the <span class="caps">IAR</span> one, this strategy optimizes the number of endorsements included in all blocks (priority <span class="math">\(0\)</span> or stolen) according to the <span class="math">\(RSD\)</span> criterion.</p>
</li>
</ul>
<h2 id="methodology-of-reward-function-evaluation"><a class="toclink" href="#methodology-of-reward-function-evaluation"><a name="sec:methodology"></a> Methodology of reward function evaluation</a></h2>
<p>Our goal is to determine how non-cooperative strategies deviate from cooperativeness as a function of the stake of a baker following that strategy, assuming the rest of the stake is held by cooperative bakers. Deviation from cooperativeness is measured according to a family of indicators derived from the profitability notions described previously: <span class="math">\(AR, IAR\)</span> and <span class="math">\(RSD\)</span>.</p>
<p>We independently applied two distinct methodologies to analyze non-cooperative strategies and design our new reward function. This allows us to be fairly confident in the validity of the results we’re about to describe.</p>
<ul>
<li>
<p>In the first approach, we designed a classical probabilistic model of the baking game to compute (either exactly or numerically) the expected values of the indicators described above, as well as some others. However, we analyzed block stealing and deflationary baking separately. This approach will be presented in detail in a follow-up post.</p>
</li>
<li>
<p>The second approach was to implement a <em>generative</em> probabilistic model of the baking game, allowing to simulate the behavior of bakers in various contexts. The programmatic nature of the simulator allows to easily plug in new strategies and new reward functions. The statistics of interest were then obtained by sampling a large number of chains averaging over the empirical distributions obtained this way.</p>
</li>
</ul>
<p>The result of this analysis led us to choose a new reward function, whose design points are the following:</p>
<ul>
<li>To make deflationary baking irrational, for all profitability criteria, the reward for including an endorsement is set to the same amount as the reward to have one endorsement included. In other words, the function makes a non-cooperative baker lose as many rewards per censored endorsement as the endorser who loses the endorsement. This property holds only for priority zero. Nevertheless, this is sufficient, because the function is also robust against block stealing, and thus most blocks should be at priority zero. The value is 1.25, or <span class="math">\(\frac{1}{32}\times 40\)</span>, giving 40 to the baker and 40 to the endorsers. Note that this will increase the variance of rewards, without impact on the total expected rewards. Concretely, a baker with 100 rolls currently has 57600 ꜩ annual expected reward with an annual standard deviation of 520 ꜩ. With the new reward function, this standard deviation would roughly double at ~1000 ꜩ.</li>
<li>To make block stealing less profitable, and since the previous point needs to equalize the rewards for the baker and the endorsers of a block, the reward for baking at priority one is set to much less than the reward for baking at priority zero. The value is <span class="math">\(\frac{6}{32}\)</span> per included endorsement. The decreasing factor is greater than in Emmy+B, but it is worth the robustness that it gives, as bakes at priority one are bonus tokens anyway.</li>
<li>On the other hand, the endorsement rewards for endorsements included in priority zero blocks are decreased by a lesser denominator than in Emmy+B (the value is 1.5 instead of 2). This does decrease slightly resistance to block stealing because the baker that steals a block gets a higher reward for his own endorsements, but has the advantage of punishing the endorsers less for having their endorsements not included by absent priority zero bakers.</li>
<li>All these constants have been chosen to be compatible with the unchanged delay function, as we wanted to keep the extra strong robustness about fork attacks that it brings, as analysed in our blog post on Emmy+.</li>
</ul>
<p>The final formulas for Emmy+C are as follows. For a block baked at priority <span class="math">\(p\)</span> and containing <span class="math">\(e\)</span> endorsements, the reward is computed as:</p>
<div class="highlight"><pre><span></span><code><span class="nv">block_reward</span><span class="w"> </span><span class="ss">(</span><span class="nv">p</span>,<span class="w"> </span><span class="nv">e</span><span class="ss">)</span><span class="w"> </span><span class="o">=</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nv">p</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="ss">(</span><span class="nv">e</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">32</span><span class="ss">)</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">80</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="ss">(</span><span class="mi">1</span><span class="o">/</span><span class="mi">2</span><span class="ss">)</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="ss">(</span><span class="nv">e</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">32</span><span class="ss">)</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">6</span>
</code></pre></div>
<p>while the reward received for each slot endorsed at priority <span class="math">\(p\)</span> is now:</p>
<div class="highlight"><pre><span></span><code><span class="nv">endorsing_reward</span><span class="w"> </span><span class="ss">(</span><span class="nv">p</span><span class="ss">)</span><span class="w"> </span><span class="o">=</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nv">p</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="ss">(</span><span class="mi">1</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">32</span><span class="ss">)</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">80</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="ss">(</span><span class="mi">1</span><span class="o">/</span><span class="mi">2</span><span class="ss">)</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="ss">(</span><span class="mi">1</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">32</span><span class="ss">)</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">80</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="ss">(</span><span class="mi">1</span><span class="o">/</span><span class="mi">2</span><span class="ss">)</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">1</span>.<span class="mi">5</span>
</code></pre></div>
<h2 id="comparing-emmyb-and-emmyc"><a class="toclink" href="#comparing-emmyb-and-emmyc"><a name="sec:comparison"></a> Comparing Emmy+B and Emmy+C</a></h2>
<p>Let us see how this new function fares compared to Emmy+B against the non-conventional baking strategies described above. For each indicator (<span class="caps">AR</span>, <span class="caps">IAR</span>, <span class="caps">RSD</span>) we give two main plots: one for Emmy+B and one for Emmy+C, corresponding to the performance of each baking strategy playing against cooperative bakers, as a function of the stake held by the non-cooperative baker. In addition, we also plot the value of each indicator when the baker plays cooperatively. Each plot corresponds to 8 hours of simulated baking time.</p>
<p>A first observation: the <code>iar-attacker</code> and <code>rsd-attacker</code> perform exactly the same, despite optimizing apparently different criteria.</p>
<h3 id="ar-profitability"><a class="toclink" href="#ar-profitability"><span class="caps">AR</span> profitability</a></h3>
<p>Let’s start by looking at how these various strategies fare with respect to <span class="caps">AR</span> profitability in the case of Emmy+B. The graph below describes the three nonstandard strategies described previously plus a cooperative one. We observe that the cooperative strategy and the <code>ar-attacker</code> strategy clearly dominate the other ones, which is not surprising. What’s more, there seems to be no visible gain by playing the <code>ar-attacker</code>: Emmy+B is well-protected by design against these attacks.</p>
<p><img alt="AR" src="images/reward_carthage/AR.png"></p>
<p>For Emmy+C the picture is similar except for the <code>rsd/iar-attacker</code> strategy which behaves differently (still not getting any noticeable <span class="caps">AR</span> gain). The discontinuity in <code>rsd/iar-attacker</code> for Emmy+C is an emergent feature of our strategy and witnesses that it is probably not optimal starting from a non-cooperative stake value of <span class="math">\(0.4\)</span>.</p>
<h3 id="iar-profitability"><a class="toclink" href="#iar-profitability"><span class="caps">IAR</span> profitability</a></h3>
<p>For Emmy+B, the best <span class="caps">IAR</span> strategies are <code>iar-attacker</code> and <code>rsd-attacker</code> which behave identically. Both have a nonnegligible effect starting at very low attacker stake, which is clearly not a good thing. The effectiveness of both strategies peaks at around <span class="math">\(60\%\)</span> of stake: after this, the opportunities for block stealing decrease as most blocks belong to the non-cooperative baker.</p>
<p><img alt="IAR" src="images/reward_carthage/IAR.png"></p>
<p>Emmy+C displays a much better behaviour: the<code>iar/rsd-attacker</code> only becomes noticeably effective starting at roughy <span class="math">\(20-25%\)</span> of the stake.</p>
<h3 id="rsd-profitability"><a class="toclink" href="#rsd-profitability"><span class="caps">RSD</span> profitability</a></h3>
<p>Emmy+B is quite vulnerable to <span class="caps">RSD</span>-based attacks as well. <code>iar/rsd-attacker</code> perform similarly for this indicator in the case of Emmy+B and the attack is effective starting at low stakes as well.</p>
<p><img alt="RSD" src="images/reward_carthage/RSD.png"></p>
<p>On the other hand, Emmy+C makes <code>iar/rsd-attacker</code> ineffective for bakers with less than <span class="math">\(20\%\)</span> of the stake.</p>
<p>These experiments indicate that Emmy+B is not robust against bakers with notions of profitability other than <span class="caps">AR</span>, even against only moderatly complex strategies. On the other hand Emmy+C resists against these strategies for non-cooperative bakers holding a share of the stake up to roughly <span class="math">\(20-25\%\)</span>. Above this, Emmy+C still fares comparatively better to Emmy+B. Emmy+C, while theoretically more fragile against <span class="caps">AR</span>-based non-cooperative baking strategies, is consistely at least as secure as Emmy+B in our simulations.</p>
<h2 id="changes-to-the-rpc-api"><a class="toclink" href="#changes-to-the-rpc-api"><a name="sec:RPC"></a> Changes to the <span class="caps">RPC</span> <span class="caps">API</span></a></h2>
<p>With the new reward formula comes some minor changes to the <span class="caps">API</span>. This
section presents these modifications.</p>
<p>The structure of the protocol constants was <strong>modified</strong> to reflect
the new formula.</p>
<p>You may access these constants using the <span class="caps">RPC</span>
<code>/chains/main/blocks/head/context/constants</code>.</p>
<p><span class="caps">TLDR</span>: The constant <code>block_reward</code> was replaced by
<code>baking_reward_per_endorsement</code> and the constant <code>endorsement_reward</code>
was modified. The rewards constants now contain a per priority reward
value that reflects the previously described formulas.</p>
<h3 id="baking-reward-per-endorsement"><a class="toclink" href="#baking-reward-per-endorsement">Baking reward per endorsement</a></h3>
<p>The field <code>block_reward</code> is replaced by the field
<code>baking_reward_per_endorsement</code> which now contains a list of <code>mutez</code>
values.</p>
<p><code>baking_reward_per_endorsement</code> is the priority-dependent number of tokens awarded to a
block producer per endorsement included in his block. The new constants are
<code>"baking_reward_per_endorsement": [ "1250000", "187500" ]</code> meaning
that <strong>a delegate producing a block of priority <code>0</code> will be rewarded</strong>
<span class="math">\(e \times 1.25\)</span> ꜩ with <span class="math">\(e\)</span> the number of endorsement included in the
block. If a delegate produces a block at priority <code>1</code>, then the second
value will be used : <span class="math">\(e \times 0.1875\)</span> ꜩ. If the list is smaller
than the given priority, the last value will be used. Thus, for the
new proposed constants, <strong>blocks baked with priorities greater than
0</strong> will receive the same reward: <span class="math">\(e \times 0.1875\)</span> ꜩ.</p>
<p>The new constants were obtained using the reward formula:</p>
<ul>
<li>
<p>For priority 0, <span class="math">\(\frac{e}{32} \times 80 \times \frac{1}{2} = e \times 1.25\)</span></p>
</li>
<li>
<p>For priority 1 and above, <span class="math">\(\frac{1}{32} \times 6 = e \times 0.1875\)</span></p>
</li>
</ul>
<h3 id="endorsement-reward"><a class="toclink" href="#endorsement-reward">Endorsement reward</a></h3>
<p>The field <code>endorsement_reward</code> is also modified into a list of <code>mutez</code>
values. The new constant are <code>"endorsement_reward": [ "1250000",
"833333" ]</code>. The semantics is similar to
<code>baking_reward_per_endorsement</code>: <strong>a delegate endorsing a block of
priority <code>0</code> will be rewarded <span class="math">\(1,25 \times e\)</span></strong> ꜩ with <span class="math">\(e\)</span> the number
of endorsing slots attributed to the delegate for this
level. Moreover, <strong>endorsing blocks of priority greater than 0 will be
rewarded <span class="math">\(0.8333333 \times e\)</span> ꜩ</strong>.</p>
<p>The new constants were obtained using the reward formula:</p>
<ul>
<li>
<p>For priority 0, <span class="math">\(\frac{e}{32} \times 80 \times \frac{1}{2} = e
\times 1.25\)</span></p>
</li>
<li>
<p>For priority 1 and above, <span class="math">\(\frac{e}{32} \times 80 \times \frac{1}{2}
\times \frac{1}{1.5} = e \times 0.833333\)</span> (rounded to the closest mutez)</p>
</li>
</ul>
<h2 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h2>
<p>Designing a good reward function is a non-trivial task due to the open and continuously evolving nature of the Tezos protocol. However, we have solid tools in our hands to quickly adapt to the unexpected ways in which people interact with the chain. We believe the new reward function is a reasonable compromise between the existing behaviour and increased security against non-standard baking strategies. Therefore it will be part of the next Carthage proposal.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Meanwhile at Nomadic Labs #42019-12-03T15:00:00+01:002019-12-03T15:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2019-12-03:/meanwhile-at-nomadic-labs-4.html<p>Summary:</p>
<p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>This is the fourth blog post of our ‘’Meanwhile at Nomadic Labs’’ series. We will discuss what we have been working on the past month, in particular about events that we organized or participated in.</p>
<p>Babylon successfully activated around one month ago. A more detailed list of changes can …</p><p>Summary:</p>
<p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>This is the fourth blog post of our ‘’Meanwhile at Nomadic Labs’’ series. We will discuss what we have been working on the past month, in particular about events that we organized or participated in.</p>
<p>Babylon successfully activated around one month ago. A more detailed list of changes can be found in our recent <a href="https://research-development.nomadic-labs.com/babylon-proposal-injected.html">Babylon Blogpost</a>. With all of these new features, Babylon was a way larger protocol change than Athens which was mostly about changing two constants. Babylon significantly touched all parts of the protocol: voting procedure, account system, consensus, and smart contracts.
Our most recent proposal, Carthage, was announced in our blog <a href="https://research-development.nomadic-labs.com/carthage-changelog-and-testnet.html">last week</a>.
While we were excited for the new protocol proposal, we didn’t expect it to get injected only a few hours after the publication of our blog post. </p>
<p>Apart for our work on the protocol, we have also been busy on other fronts:</p>
<p>Nomadic Labs released its first teaser ”Discover Nomadic Labs in 2 minutes” to introduce ourselves to the French and international community. You can view it <a href="https://youtu.be/d6iwBXKD_js">here</a>.</p>
<p>We are equally proud to have hosted our first ever Pitch Day.
Six blockchain startups from different industries had the opportunity to pitch in front of five experts. You can view the participants in this <a href="https://twitter.com/LabosNomadesFR/status/1192859842077167617?s=20">tweet</a>.
<a href="https://www.equisafe.io/">Equisafe</a>, a company that specializes in tokenizing real-estate assets on the blockchain, won the competition. They also used the opportunity to announce their first smart contract on the Tezos blockchain (one of the four that will compose the core components of the <span class="caps">NYX</span> Standard, which is planned for 2020)</p>
<p>We were the main sponsor of Chainhack 4, a mini hackathon in Lisbon between the 8th and 10th of November.
Together with Tezos Commons, we are also sponsoring the upcoming <a href="https://gettogether.community/events/3010/barcelona-tezos-meetup/">Tezos Barcelona meetup</a>.</p>
<p>Raphaël Cauderlier, Julien Tesson and Colin González from Nomadic Labs presented at the ‘’<a href="http://michel.mauny.net/misc/2019-11-tezos-workshop/">Tezos Smart Contracts: Programming Languages and Formal Verification</a>’’ workshop hosted by <a href="https://www.irif.fr/"><span class="caps">IRIF</span></a>, the fundamental computer science laboratory of “Université de Paris” (formerly known as “Université Paris-Diderot”). This informal workshop, sponsored by Nomadic Labs and organized by Michel Mauny from the Tezos Foundation, introduced some of the available programming languages, some more experimental languages, and some of the formal verification techniques that are being developed in the Tezos ecosystem.
Raphaël gave an introduction to Michelson and a presentation of <a href="https://gitlab.com/nomadic-labs/mi-cho-coq">Mi-Cho-Coq</a>, whereas Julien talked about <a href="https://albert-lang.io/">Albert</a>, an Intermediate Language for Tezos Smart Contracts. Colin presented his PhD thesis project: a spreadsheet-derived language for smart-contract programming.
The entire workshop was live streamed in two parts thanks to Marc-Antoine Tréhin:
<a href="https://youtu.be/7AX0XrBC2BA">#1</a>
<a href="https://youtu.be/TEMRtzCu5kk">#2</a>.</p>
<p>We also attended <a href="https://blockchainevent.fr/2019/">Blockchain Corp Paris 2019</a>, with a dedicated Tezos booth. (A large Exhibit with +1500 participants focused on Enterprise and Adoption).
Hadrien Zerah, Sajida Zouarhi and Sebastian Larquié were present during this two-day conference to talk about the entire Tezos ecosystem, the challenges ahead, and what differentiates Tezos from many of the other projects out there. We also held a presentation in the workshop area and answered R&D/adoption-related questions.</p>
<p>Arvid Jakobsson and Zaynah Dargaye are working on formally verifying the <a href="https://research-development.nomadic-labs.com/cortez-security-by-using-the-spending-limit-contract.html">Spending Limit Contract</a>,
using <a href="https://gitlab.com/nomadic-labs/mi-cho-coq">Mi-Cho-Coq</a>. This contract limits the amount of spending that an account can perform in a configurable time
window. This is similar to how a typical credit card imposes daily
spending limits for security reasons. This contract forms an integral
part of the <a href="https://gitlab.com/nomadic-labs/cortez-android">Cortez wallet</a>,
making its formal verification a priority.</p>
<p>Last but not least, we are currently looking for interns! Apply before December 15th. More details in this <a href="https://twitter.com/LabosNomades/status/1197836576245596160?s=20">tweet</a>.</p>Carthage: changelog and testnet2019-11-14T18:00:00+01:002019-11-14T18:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2019-11-14:/carthage-changelog-and-testnet.html<p>Summary:</p>
<p>We are happy to announce a new protocol proposal code-named Carthage,
developed together with <a href="https://medium.com/cryptium">Cryptium Labs</a>.</p>
<p>After the feature rich Babylon, Carthage is a much needed housekeeping
release.
Carthage contains exclusively fixes and small improvements across the
board, the more relevant being a corrected formula for rewards.
The only …</p><p>Summary:</p>
<p>We are happy to announce a new protocol proposal code-named Carthage,
developed together with <a href="https://medium.com/cryptium">Cryptium Labs</a>.</p>
<p>After the feature rich Babylon, Carthage is a much needed housekeeping
release.
Carthage contains exclusively fixes and small improvements across the
board, the more relevant being a corrected formula for rewards.
The only “novelty” is an increase in the gas limit which will allow to
execute more smart contracts.</p>
<p>A testnet called
<a href="https://tezos.gitlab.io/introduction/test_networks.html#carthagenet"><code>carthagenet</code></a>
has already been running Babylon since last week and will amend to
Carthage in the next days to simulate a protocol migration.
We encourage users and developers to join the testnet and the discussion
on the tezos-backing slack.</p>
<p>As usual refer to the <a href="http://tezos.gitlab.io/protocols/006_carthage.html">protocol documentation
page</a> for more
details and for the full
<a href="http://tezos.gitlab.io/protocols/006_carthage.html#changelog">Changelog</a>.</p>
<p>As promised in our <a href="https://research-development.nomadic-labs.com/lessons-learned-from-the-babylon-protocol-upgrade-a-retrospective.html">previous post</a>,
in order to give the community time to review and discuss this
proposal, we are not injecting it ourself but we are simply providing
the code.
Any baker supportive of this proposal can inject it following the
instructions below.</p>
<h1 id="getting-the-source-code-of-the-proposals"><a class="toclink" href="#getting-the-source-code-of-the-proposals">Getting the source code of the proposals</a></h1>
<p>The source code of this proposal is available in
<a href="files/carthage_006_PtCartha.tar">this tar archive</a>.</p>
<p>To compute the hash, you can use the <code>tezos-protocol-compiler</code> as
follows. The result should be <code>PtCarthavAMoXqbjBPVgDCRd5LgT7qqKWUPXnYii3xCaHRBMfHH</code>.</p>
<div class="highlight"><pre><span></span><code>mkdir proposal && cd proposal
curl https://research-development.nomadic-labs.com/files/carthage_006_PtCartha.tar | tar -x
tezos-protocol-compiler -hash-only carthage_006_PtCartha/lib_protocol
</code></pre></div>
<p>If you want to directly investigate how Carthage differs from the current
protocol, you can use <code>diff</code> against folder <code>proto_005_PsBabyM1</code> of the
mainnet source code. All the changes should be the ones detailed in the <em>changes</em>
section, except for the removal of the single use migration code from
the previous upgrade (from <code>004_Pt24m4xi</code> to <code>005_PsBabyM1</code>).</p>
<h1 id="injecting-the-proposal"><a class="toclink" href="#injecting-the-proposal">Injecting the proposal</a></h1>
<p>Any active baker can inject the proposal by simply downloading the
code, injecting it in a running mainnet node with <code>tezos-admin-client</code>
and using the <code>submit</code> command of <code>tezos-client</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-admin-client<span class="w"> </span>inject<span class="w"> </span>protocol<span class="w"> </span>carthage_006_PtCartha/lib_protocol
Injected<span class="w"> </span>protocol<span class="w"> </span>PtCarthavAMoXqbjBPVgDCRd5LgT7qqKWUPXnYii3xCaHRBMfHH<span class="w"> </span>successfully
$<span class="w"> </span>tezos-admin-client<span class="w"> </span>list<span class="w"> </span>protocols
Ps9mPmXaRzmz
PsddFKi32cMJ
PsYLVpVvgbLh
PtCJ7pwoxe8J
Pt24m4xiPbLD
PsBabyM1eUXZ
PtCarthavAMo<span class="w"> </span><------
$<span class="w"> </span>tezos-client<span class="w"> </span>submit<span class="w"> </span>proposals<span class="w"> </span><span class="k">for</span><span class="w"> </span><delegate><span class="w"> </span>PtCarthavAMo
</code></pre></div>Lessons Learned from the Babylon Protocol Upgrade: A Retrospective2019-10-30T18:00:00+01:002019-10-30T18:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2019-10-30:/lessons-learned-from-the-babylon-protocol-upgrade-a-retrospective.html<p>Summary:</p>
<p>Babylon (aka protocol <code>005</code>), the second Tezos protocol amendment jointly developed by <a href="https://nomadic-labs.com">Nomadic Labs</a> and <a href="https://twitter.com/CryptiumLabs">Cryptium Labs</a>, was successfully activated on block <code>655361</code>. Since then, we’ve continued analysing and monitoring the new features, but have also engaged in a deeper reflection on the upgrade process from its development …</p><p>Summary:</p>
<p>Babylon (aka protocol <code>005</code>), the second Tezos protocol amendment jointly developed by <a href="https://nomadic-labs.com">Nomadic Labs</a> and <a href="https://twitter.com/CryptiumLabs">Cryptium Labs</a>, was successfully activated on block <code>655361</code>. Since then, we’ve continued analysing and monitoring the new features, but have also engaged in a deeper reflection on the upgrade process from its development period, pre-injection, to the period following the activation.</p>
<p>This article summarises the lessons learned in five parts: the development of Babylon, the proposal period, continued testing during the on-chain test period, activation, and post-activation. For each of these parts we identify the missteps made over the past months and draw lessons for current and future core developers to improve on the process for future protocol upgrades.</p>
<p>The process of developing Babylon had many firsts. It was the first time that two independent core development teams worked together on a proposal, which was a non-trivial step towards decentralisation of core development. Furthermore, given its large set of features, it was a steep step from Athens. While Athens proved that live upgrades worked, Babylon showed that large parts of the codebase can be amended in order to deliver meaningful improvements.</p>
<p><strong><span class="caps">TL</span>;<span class="caps">DR</span></strong> : Less rush, more testing, more documentation, more community involvement.</p>
<h2 id="developing-babylon"><a class="toclink" href="#developing-babylon">Developing Babylon</a></h2>
<h3 id="issue-controlling-the-set-of-changes-and-timeline"><a class="toclink" href="#issue-controlling-the-set-of-changes-and-timeline">Issue: controlling the set of changes and timeline</a></h3>
<p>An issue during the development of Babylon was that, despite agreeing beforehand on a set of features, we ended up adding a few more along the way. This was due partly to some backlog from Athens and partly because we received requests for features that we were afraid to leave unanswered for 3 more months. Furthermore we were trying to respect a specific timeline for injection because we were afraid of a never ending development cycle where changes kept piling up.</p>
<p>Working towards a moving target on a fixed timeline complicated considerably the internal development and rushed the final stages of the release, impacting testing and the communication with the community.</p>
<h3 id="how-to-improve"><a class="toclink" href="#how-to-improve">How to improve</a></h3>
<p>In future upgrades we will compile a list of features early on and gather input from the wider ecosystem. We will limit development towards the specific features and this will give us a clearer vision of a possible timeline that, if needed, will be delayed until proper review, testing and communication is done.</p>
<h2 id="proposal-period-for-babylon"><a class="toclink" href="#proposal-period-for-babylon">Proposal Period for Babylon</a></h2>
<p>The first proposal (Babylon <code>PsBABY5n</code>) was injected on the 26th of July 2019. Shortly after this, a new updated version (Babylon <code>PsBABY5H</code>) was proposed during the same proposal period.</p>
<h3 id="issue-binary-format-of-endorsements"><a class="toclink" href="#issue-binary-format-of-endorsements">Issue: Binary format of endorsements</a></h3>
<p>The second injection was in response to feedback we received from wallet developers which led us to revert a change in the binary format of endorsements. The binary format of manager transactions was kept in order to ensure that user transaction from <code>004</code> could not be executed on <code>005</code>.</p>
<p>The proposal phase was indeed conceived with the idea that there could be a series of proposals, counter proposals and iterations.</p>
<p>However, this extra injection implied more off-chain coordination, and it could have been avoided if we had gathered feedback from developers impacted by the binary format of operations before the first proposal.</p>
<h3 id="how-to-improve_1"><a class="toclink" href="#how-to-improve_1">How to Improve</a></h3>
<p>We see three areas of improvement:</p>
<ol>
<li>
<p>Ensure that the binary formats are well documented, before the injection</p>
<p>All the binary formats of the protocol can be obtained with the new binary <code>tezos-codec</code> for any of the encodings used in the Tezos codebase. We are also working on a new tool that will parse and display the binary format of a specific operation.</p>
<p>Any breaking change introduced by a proposal is highlighted in the <a href="http://tezos.gitlab.io/protocols/005_babylon.html#changelog">Changelog</a> that we publish together with every release and there are specific guidelines that are suggested to handle the <a href="http://tezos.gitlab.io/protocols/005_babylon.html#changes-to-the-binary-format-of-operations">migration</a>.</p>
<p>We will continue improving the quality and accessibility of this documentation and we will make sure that it is available to third party developers a few weeks before a proposal.</p>
</li>
<li>
<p>Allow developers to test properly, before the injection</p>
<p>Using the simple <a href="http://tezos.gitlab.io/user/sandbox.html">bash sandbox</a> is usually enough to interact with a Tezos node and client and rapidly test by hand. Additionally a number of more complex end-to-end tests can be automated using one of the two frameworks that are present in the code base, <a href="http://tezos.gitlab.io/developer/flextesa.html">Flextesa</a> and the <a href="http://tezos.gitlab.io/developer/python_testing_framework.html">Python test framework</a>.</p>
<p>These frameworks are used by core developers, which means that they will be maintained in the future and that there are a number of existing tests that can serve as inspiration to third party developers. We believe that with the tools presented any future incompatibility in the format of operations can be detected early in the development process and independently, without the need for a testchain.</p>
<p>However, these tools might not have been advertised enough to third party developers, and we will make an effort to better document them and encourage their adoption and feedback.</p>
</li>
<li>
<p>Publish the code and changelog before the injection</p>
<p>For Babylon we published a description of the features as well as links to their implementation as we build them over time, for example <a href="https://medium.com/cryptium/meanwhile-at-cryptium-labs-1-part-ii-607227fc6d65">Quorum Caps</a> and <a href="https://medium.com/cryptium/meanwhile-at-cryptium-labs-1-part-iii-1c824b760da3">Making implicit accounts delegatable</a>.</p>
<p>However for both Athens and Babylon the changelog and final code was only published at the same time as the proposal was injected. In the future we will make sure to leave a few weeks of time between publication and injection to leave ample time for last minute corrections.</p>
</li>
</ol>
<h2 id="exploration-testing-periods-for-babylon"><a class="toclink" href="#exploration-testing-periods-for-babylon">Exploration <span class="amp">&</span> Testing Periods for Babylon</a></h2>
<h3 id="issue-the-issue-with-big-maps-and-the-hotfix"><a class="toclink" href="#issue-the-issue-with-big-maps-and-the-hotfix">Issue: the issue with Big Maps and the Hotfix</a></h3>
<p>During the test period of Babylon <code>PsBABY5H</code> we discovered a bug affecting big maps. More details are in the <a href="http://tezos.gitlab.io/protocols/005_babylon.html#bug-affecting-bigmaps-in-005-psbaby5h">documentation</a>. The bug was particularly hard to catch because it was a single line in one large patch and because it didn’t break any functionality but caused a performance degradation.</p>
<p>Despite the above we believe that with a more rigid review process and more automated tests we should be able to avoid this situations in the future.</p>
<p>The natural course of action would have been to vote <code>nay</code> in the promotion vote and to propose a patched version of Babylon, <code>PsBabyM1</code>, incurring a delay of 3 months for the voting procedure to complete again.</p>
<p>However, given that multiple big maps (together with entry-points) were strongly requested by smart contract developers, we felt a sense of urgency from some parts of the community to deliver this feature as soon as possible.</p>
<p>For this reason we decided to add the possibility to download a new version of the Tezos node that in case of a positive vote for Babylon <code>PsBABY5H</code> would activate the patched version <code>PsBabyM1</code>, explained in the <a href="https://research-development.nomadic-labs.com/mainnet-release-to-patch-babylon.html">related blog post</a>.</p>
<p>Adding this possibility was not meant as a suggestion, but merely as an additional option for the community. The majority of nodes preferred this option and <code>PsBabyM1</code> was activated successfully.</p>
<p>We have always considered user-activated updates a legitimate part of the governance process when used for urgent and critical hotfixes, like we did in the past and as explained in the <a href="https://research-development.nomadic-labs.com/amendments-at-work-in-tezos.html">“Amendments at work” blog post</a>.</p>
<p>On the positive side, this hotfix allowed to unlock the many new possibilities given by big maps. We are preparing a blog post on the topic for smart contract authors.</p>
<h3 id="how-to-improve_2"><a class="toclink" href="#how-to-improve_2">How to Improve</a></h3>
<p>There were several issues at play here.</p>
<p>To allow for faster iteration over multiple improved versions of a proposal, we’ve discussed several ways to reduce the length of a voting procedure in case of a negative vote.</p>
<p>The hot fix also led us to reflect on the user-activated update mechanism. Because of a technical limitation in the current implementation, user-activated updates can only be introduced by releasing new binaries or recompiling the code, a process which can leave non developers out. We are implementing a node where user-activated updates can be set easily by users in a config file, instead of depending on a mainnet release.</p>
<p>As for the big map itself, this kind of bug can be found through more thorough testing and stricter review requirements during the development process. As the number of developers competent in protocol development grows so will the number of reviews per merge request.</p>
<p>In addition, more developer activity on the testnets prior to injection would be helpful. One way to achieve that is to look into incentivizing participation in the testnet. Another way, which has already been enacted, consists in always having a testnet running the proposal currently being voted on.</p>
<h2 id="activation-of-babylon"><a class="toclink" href="#activation-of-babylon">Activation of Babylon</a></h2>
<h3 id="issue-the-mempool-glitch-at-activation"><a class="toclink" href="#issue-the-mempool-glitch-at-activation">Issue: the mempool glitch at activation</a></h3>
<p>Right after the activation of Babylon there was a slowdown of block production in the network due to some nodes being blocked. These nodes had correctly migrated to Babylon but were still receiving operations from Athens which they were no longer capable of deserialising, causing a failure of the mempool and of the node.</p>
<p>Fortunately the situation could easily be resolved by simply restarting the nodes, thus erasing the old operations, however, those operations would still be propagated by the network for an hour, requiring multiple restarts of the node. A patch to fix the mempool was ready minutes after we realised the problem and many bakers readily updated their nodes.</p>
<p>The failure of the mempool was caused by a bug which was known and fixed on the master branch of the Tezos code base but was incorrectly ported to the mainnet branch.</p>
<h3 id="how-to-improve_3"><a class="toclink" href="#how-to-improve_3">How to Improve</a></h3>
<p>Once again this kind of bug can be avoided using a simpler release process which we are currently implementing and will be in place before the next release. The goal is to reduce to a minimum the number of steps needed to release Mainnet starting from the master development branch. This will simplify testing and leave less room for errors.</p>
<p>In the future, we will run testnets before injection that simulate the migration from the old state machine to the new. In this instance it would have meant running a testnet with <code>004</code> and then upgrading the running network to <code>005</code>, while at the same time generating load by randomly sending transactions. This kind of testnet would have caught the previous bug.</p>
<h2 id="post-activation-of-babylon"><a class="toclink" href="#post-activation-of-babylon">Post-Activation of Babylon</a></h2>
<h3 id="issue-the-formula-for-block-rewards"><a class="toclink" href="#issue-the-formula-for-block-rewards">Issue: the formula for block rewards</a></h3>
<p>After the activation of Babylon, users realised that the computation of rewards was slighly different with respect to the formula published in the blog post describing <a href="https://research-development.nomadic-labs.com/emmy-an-improved-consensus-algorithm.html">Emmy+</a>, the <a href="https://research-development.nomadic-labs.com/analysis-of-emmy.html">improved</a> consensus algorithm.</p>
<p>The reason for this difference is a bug in the implementation of the formula, which results in a loss of precision when calculating the baking rewards.</p>
<p>This precision loss does not affect the security of Emmy+ and a fix will be offered in the upcoming protocol upgrade.</p>
<h3 id="how-to-improve_4"><a class="toclink" href="#how-to-improve_4">How to Improve</a></h3>
<p>This kind of bugs should be more easily spotted and avoided, we see two lessons to learn here.</p>
<ol>
<li>
<p>Can we write fewer bugs in the future?</p>
<p>We are going to improve our release process for protocols by having</p>
<ul>
<li>comprehensive unit tests</li>
<li>a clearer, stricter, policy for code reviews</li>
<li>a clearer freeze and review period before injection</li>
</ul>
</li>
<li>
<p>Bugs still happen, how can we make sure we notice them?</p>
<p>Despite having Babylonnet running since September 27th, we didn’t sufficiently encourage participation and for this reason the traffic on the network has not been a realistic sample of mainnet.</p>
<p>Furthermore the lack of testnet support from block explorers has strongly limited the capacity of users and developers to inspect the data of the testnet.</p>
<p>In the last month there was a reorganisation of the testnets infrastructure to improve the service and engage more members of the community.</p>
<p>Moreover several block explorers, such as <a href="https://tzstats.com/">TzStats</a>, are working to support Babylonnet and will support future test chains.</p>
</li>
</ol>
<h3 id="issue-restricting-originated-kt1-contracts-from-paying-transaction-fees"><a class="toclink" href="#issue-restricting-originated-kt1-contracts-from-paying-transaction-fees">Issue: restricting originated (<span class="caps">KT1</span>) contracts from paying transaction fees</a></h3>
<p>With the implementation of the delegation process simplification, originated accounts (<span class="caps">KT1</span>) can no longer pay for transaction fees. The goal of this change is to ensure that all transaction fees are always paid by tz1 addresses, and remove the computational overhead produced by fees paid through <span class="caps">KT1</span> accounts, as smart contracts need to be fully executed in order to verify their validity. This results in the potential for dramatic mempool optimisations and increased throughput.</p>
<p>However, the now legacy multi-step delegation process led to a common scenario, where all the funds of the tz1 account were transferred to the <span class="caps">KT1</span> account to maximise the amount delegated. Before Babylon, this was not an issue as the <span class="caps">KT1</span> account was able to pay for transaction fees.</p>
<p>With Babylon and <span class="caps">KT1</span> accounts no longer able to pay for transaction fees, implicit accounts that had empty balances before the protocol upgrade where funded with 1µꜩ (0.000001ꜩ) to prevent the requirement for a new allocation burn.</p>
<p>This led to two issues. First, the creation of these accounts and their funding with 1µꜩ was not documented, which led to trouble for block explorers. Second, the 1µꜩ balance was not sufficient to pay for a transaction.</p>
<p>To assist affected accounts, Cryptium Labs funded all the implicit accounts in this situation with 0.01ꜩ, which is high enough for the account to pay for at least one transfer transaction (funding, for instance, the tz1 address with enough fees to pay for several transactions).</p>
<h3 id="how-to-improve_5"><a class="toclink" href="#how-to-improve_5">How to Improve</a></h3>
<p>Regarding the effect of token creation during protocol migration and the way they are interpreted by block explorers, we feel the cleanest, most consistent, approach is to introduce a receipt attached to migration blocks showing all of the balance updates caused by the migration. This however, requires a change in the Tezos environment, and not merely a protocol change. Protocol environments are versioned and designed to change over time (for instance to accomodate new cryptographic libraries).</p>
<p>Regarding the effect of the transaction fee, funding the accounts directly outside of the protocol upgrade was a simple lo-tech solution which worked. However, if anything of the sort were to be repeated in the future, it should involve clearer communication with the affected users in order to ensure a smooth upgrade for everyone.</p>
<h2 id="things-that-worked"><a class="toclink" href="#things-that-worked">Things that Worked</a></h2>
<p>This is the first time that a running decentralised, permissionless and censorship resistant blockchain protocol evolved in a <em>meaningful</em> way. Babylon paves the way for a chain that can evolve over time and adapt the best technologies from the entire ecosystem.</p>
<p>Furthermore, it is the first time, for Tezos, that two independent core development teams worked on the same protocol. Hiccups were abundant, but it worked and it showed that core development can be decentralised while also moving fast and evolving the protocol.</p>
<p>Lastly, let’s not forget the many features that worked:</p>
<ul>
<li><a href="https://medium.com/cryptium/meanwhile-at-cryptium-labs-1-part-ii-607227fc6d65">Quorum caps</a></li>
<li><a href="https://research-development.nomadic-labs.com/emmy-an-improved-consensus-algorithm.html">Emmy+</a></li>
<li><a href="https://medium.com/cryptium/meanwhile-at-cryptium-labs-1-part-iii-1c824b760da3">Making implicit accounts delegatable</a></li>
<li><a href="https://research-development.nomadic-labs.com/michelson-updates-in-005.html">Multiple big_maps and entrypoints</a></li>
<li><a href="https://medium.com/cryptium/meanwhile-at-cryptium-labs-1-part-v-3e0ddfd98177">Spam prevention for proposals</a></li>
</ul>
<h2 id="closing-remarks"><a class="toclink" href="#closing-remarks">Closing Remarks</a></h2>
<p>In summary, we have identified the following as key areas to improve the proposal process:</p>
<ul>
<li>Outline the desired features ahead of time, stricly focus development on them and wait all reviews, testing and documentation to be done.</li>
<li>Make feature documentation and changelogs more accessible and visible ahead of time, to give the community more time to engage.</li>
<li>Release of independent features to testnets, so ecosystem developers have an early access and can provide feedback in advance.</li>
<li>Reorganisation and maintainance of testnets.</li>
<li>More reviews per merge requests and more unit and integration testing.</li>
</ul>
<p>Lastly it is important to remember the scope of these changes for Tezos. Babylon is, feature-wise, a steep step from Athens, which touched almost all the main areas of the protocol: Michelson, voting procedure, accounts and consensus. With the Babylon upgrade the Tezos community proved to the world that we are the first blockchain that can significantly amend a running protocol. Although there were more than a few drawbacks, it should not deter us from improving Tezos over time, as now is the time to create the foundation for a long lasting and relevant blockchain protocol.</p>Mainnet release to patch Babylon2019-09-20T18:00:00+02:002019-09-20T18:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2019-09-20:/mainnet-release-to-patch-babylon.html<p>During the testing phase of protocol Babylon <code>005_PsBABY5H</code>, we
discovered a bug affecting bigmaps in Michelson smart contracts.
The bug is corrected in a new version of Babylon, <code>005_PsBabyM1</code>.</p>
<h1 id="how-to-proceed"><a class="toclink" href="#how-to-proceed">How to proceed</a></h1>
<p>The bug is not critical but causes a significant performance
degradation for newly created smart contracts and an incorrect behavior
for existing ones using big maps.
More details on the nature and effects of the bug can be found in the
<a href="http://tezos.gitlab.io/protocols/005_babylon.html#bug-affecting-bigmaps-in-005-psbaby5h">protocol documentation
page</a>.</p>
<p>During the upcoming promotion vote period, we propose two options.
The first obvious possibility is to vote Nay so that the current
Athens protocol will proceed to a new proposal period.
Protocol <code>005_PsBabyM1</code> will then be proposed and we will continue with a
normal voting procedure.
As usual the voting procedure will last 3 months if each
voting period is successful.</p>
<p>The second option is for users to download a new version of the Tezos
node that, in case of a positive promotion vote, will activate the
corrected protocol Babylon <code>005_PsBabyM1</code>, instead of the defective
<code>005_PsBABY5H</code>.
Note that in case of a negative vote this new release of the Tezos
node will simply continue to run Athens as expected.</p>
<p>The advantage of the second option is to save the time of a full
voting procedure to simply fix a bug.
However it is up to the community to decide which option to take.</p>
<p>We do our best to prevent bugs from slipping inside our releases but
unfortunately, as with any complex software, there is always a small chance of missing something.
This episode proves once again the importance of having a thorough
voting and testing procedure.</p>
<h1 id="the-patched-protocol-babylon-005_psbabym1"><a class="toclink" href="#the-patched-protocol-babylon-005_psbabym1">The patched protocol Babylon <code>005_PsBabyM1</code></a></h1>
<p>As with any protocol release it is important to inspect the sources
behind a hash.
In this case the diff with respect to <code>005_PsBABY5H</code> consists of two
lines, one to fix the bigmap bug described above and one to fix an
unrelated problem in an <span class="caps">RPC</span> call.</p>
<p>The source of the new Babylon can be found in <a href="files/babylon_005_PsBabyM1.tar">this tar
archive</a> and as usual you can
recompute the hash using</p>
<div class="highlight"><pre><span></span><code>mkdir proposal && cd proposal
curl https://research-development.nomadic-labs.com/files/babylon_005_PsBabyM1.tar | tar -x
tezos-protocol-compiler -hash-only babylon_005_PsBabyM1/lib_protocol
</code></pre></div>
<p>The result should be <code>PsBabyM1eUXZseaJdmXFApDSBqj8YBfwELoxZHHW77EMcAbbwAS</code>.</p>
<div class="highlight"><pre><span></span><code><span class="gh">diff --git a/src/proto_alpha/lib_protocol/script_ir_translator.ml b/src/proto_alpha/lib_protocol/script_ir_translator.ml</span>
<span class="gh">index 5ad517ab4..b73d610ba 100644</span>
<span class="gd">--- a/src/proto_alpha/lib_protocol/script_ir_translator.ml</span>
<span class="gi">+++ b/src/proto_alpha/lib_protocol/script_ir_translator.ml</span>
<span class="gu">@@ -1038,7 +1038,7 @@ let has_big_map</span>
<span class="w"> </span> | Bool_t _ -> false
<span class="w"> </span> | Lambda_t (_, _, _) -> false
<span class="w"> </span> | Set_t (_, _) -> false
<span class="gd">- | Big_map_t (_, _, _) -> false</span>
<span class="gi">+ | Big_map_t (_, _, _) -> true</span>
<span class="w"> </span> | Contract_t (_, _) -> false
<span class="w"> </span> | Operation_t _ -> false
<span class="w"> </span> | Chain_id_t _ -> false
<span class="gh">diff --git a/src/proto_alpha/lib_protocol/script_interpreter.ml b/src/proto_alpha/lib_protocol/script_interpreter.ml</span>
<span class="gh">index 86d5dde53..3e4917b1a 100644</span>
<span class="gd">--- a/src/proto_alpha/lib_protocol/script_interpreter.ml</span>
<span class="gi">+++ b/src/proto_alpha/lib_protocol/script_interpreter.ml</span>
<span class="gu">@@ -978,7 +978,7 @@ and interp</span>
<span class="w"> </span> log := (code.loc, Gas.level ctxt, stack) :: !log ;
<span class="w"> </span> return_unit
<span class="w"> </span> end >>=? fun () ->
<span class="gd">- step ctxt step_constants code stack >>=? fun (Item (ret, Empty), ctxt) -></span>
<span class="gi">+ step ?log ctxt step_constants code stack >>=? fun (Item (ret, Empty), ctxt) -></span>
<span class="w"> </span> return (ret, ctxt)
<span class="w"> </span>(* ---- contract handling ---------------------------------------------------*)
</code></pre></div>Cortez security by using the Spending Limit contract2019-09-12T15:00:00+02:002019-09-12T15:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2019-09-12:/cortez-security-by-using-the-spending-limit-contract.html<p>In this most recent blog post we discuss how to benefit from the secure enclave using the Daily Spending Limit smart contract.</p><h2 id="my-wallet-my-crypto"><a class="toclink" href="#my-wallet-my-crypto">My wallet, my crypto</a></h2>
<p>When holding a cryptocurrency, one needs a safe and secure wallet to store it. Wallets are used <strong>to store, receive, or spend</strong> cryptocurrencies. They work by keeping private keys to themselves and using them to sign transaction from the corresponding public keys. </p>
<p>A public key is used to send assets to the corresponding destination, whereas a private key is used to send assets from your corresponding wallet. Addresses are uniquely defined as a public function of the public key.</p>
<p>The most important component of a cryptocurrency wallet is the <strong>private key</strong>: if one loses it, then the corresponding tokens are no longer accessible. Cryptocurrency holders must take really good care of where and how they store their private keys. To make things easier, one can use a <strong>seed phrase</strong>, which is a randomly generated list of words used to deterministically generate a secret key. Therefore the seed phrase gives the ability <strong>to recover an account and the corresponding funds</strong>.</p>
<p>This is typically what cryptocurrency wallet software tends to do - they randomly generate seed phrases and ask users to write them down on a piece of paper. In the unfortunate event that something were to happen, like losing a phone or breaking a computer, a user can download the wallet software on a different device and use a seed phrase to recover the associated account.</p>
<h2 id="keep-it-safe-stupid"><a class="toclink" href="#keep-it-safe-stupid">Keep it safe, stupid<a href="#external-links">*</a></a></h2>
<p>If one would rather not leave their private key on a mobile wallet, a <strong>hardware wallet</strong> is a good alternative. This little gadget has no network capabilities, for obvious security reasons, and therefore must be used in conjunction with a computer/phone in order to transact.</p>
<p>Even so, some phones use the same level of security one can expect from a hardware wallet like a <a href="https://shop.ledger.com/products/ledger-nano-s">Ledger Nano S</a>. <strong>Apple</strong> and its <a href="https://developer.apple.com/documentation/security/certificate_key_and_trust_services/keys/storing_keys_in_the_secure_enclave">secure enclave</a> and <strong>Samsung</strong>, which relies on <a href="https://developer.arm.com/ip-products/security-ip/trustzone"><span class="caps">ARM</span> TrustZone</a> technology, both provide hardware isolation (“hardware security modules”) to prevent malicious software from extracting private keys.</p>
<h2 id="peg-leg-pete-got-me"><a class="toclink" href="#peg-leg-pete-got-me">Peg-Leg Pete got me</a></h2>
<p>You might feel safe using a private key, but what if someone like <strong>Peg-Leg Pete</strong> asks you for your cryptocurrency wallet in a way that you can’t refuse? What if he commands you to transfer all your cryptocurrencies to him, or else… How unfortunate it would be to lose everything despite taking so much care of keeping your private key safe!</p>
<p><img alt="pete" src="images/slc/pete.jpg"></p>
<p><em>Peg-Leg Pete threatens Mickey © Disney</em></p>
<p>A similar situation with a credit card would not have been as critical. At most you would have lost what your daily limit allows for before calling your bank to file a report/chargeback.</p>
<h1 id="cortez-proposal"><a class="toclink" href="#cortez-proposal">Cortez proposal</a></h1>
<h2 id="the-secure-enclave"><a class="toclink" href="#the-secure-enclave">The Secure enclave</a></h2>
<p>A few words about the <strong>secure enclave</strong> on mobile devices.</p>
<p>Some smartphones (iOS/Android) possess special security chips (<span class="caps">HSM</span>) which are basically tiny computers themselves. One of these HSMs has its own operating system running on top of its own processor and memory. With this, an attacker will likely never be able to access the contents of this secure area even if he/she is able to compromise the main operating system of the phone.</p>
<p>When using Apple Pay, Google Pay and/or Samsung Pay, payment details are stored securely in an <span class="caps">HSM</span> to prevent access by malicious software.</p>
<p><img alt="Enclave" src="images/slc/enclave.png"></p>
<p><a href="https://www.researchgate.net/publication/318402496_Establishing_Mutually_Trusted_Channels_for_Remote_Sensing_Devices_with_Trusted_Execution_Environments">Secure enclave inside a mobile phone</a></p>
<p>Something really interesting about the Trusted World (referenced above) on mobile is that it allows users to generate cryptographic keys and encrypt/decrypt/sign/verify and so on, <strong>without private keys being exposed</strong>. Private keys stay in the secure enclave, the Trusted World.</p>
<p>Signing Tezos transactions with a key generated in a secure enclave would be perfect to <strong>avoid the threat of malicious actors</strong>.
It’s actually already possible to do that.</p>
<p>Mobile operating systems like iOS and Android allow us to generate a <strong><span class="caps">NIST</span> P-256 (aka secp256r1)</strong> key pair for <span class="caps">ECDSA</span> and store the private key in the Trusted World.
This elliptic curve is the one used to generate <strong>tz3 addresses</strong> for Tezos accounts.</p>
<p>We could then have a tz3 public key hash and <strong>sign transactions using the corresponding secure enclave</strong>, that would be perfect. </p>
<p>The problem with secure enclave key generations is that you cannot extract a private key and use it later on via another device. You can sign, verify, encrypt, and decrypt data, but exports are not allowed.</p>
<p>This is the whole point of a secure enclave - <strong>Nobody ever has access to a private key itself</strong>. If you lose your phone with this configuration, the wallet on your phone (and all its contents) is <strong>lost forever inside the phone’s secure enclave</strong>.</p>
<h2 id="secure-enclave-using-a-smart-contract"><a class="toclink" href="#secure-enclave-using-a-smart-contract">Secure enclave using a smart contract</a></h2>
<p>Our goal is to use a phone’s secure enclave to generate a tz3 address to be able to sign tezos transactions with it, while at the same time facilitating a great user experience to enable actions such as account recovery. For this, we can use a smart contract.</p>
<p>The contract is created from a main account (tz1…) simply by <strong>originating a <span class="caps">KT1</span> account with the <em>Spending Limit Contract</em> code</strong>, preferably with some ꜩ in it.</p>
<p>The visual below summarizes the contract:</p>
<p><img alt="Schema" src="images/slc/schema.png"></p>
<p>The contract keeps two pieces of information in its storage: </p>
<ul>
<li>a public key hash (tz3…), aka the “signatory.”</li>
<li>a spending limit (explained in the next paragraph).</li>
</ul>
<p>The contract will send transactions from the <span class="caps">KT1</span> account to another address if, and only if, <strong>the transaction is signed using the secure enclave private key</strong> used to generate the tz3 in its storage.</p>
<p>The <em>Spending Limit Contract</em> storage is <strong>amendable using a 24-word mnemonic (seed phrase)</strong>. Thus, it is possible to change a contract’s public key hash and spending limit.</p>
<p><em>Once again, keep your mnemonic/seed phrase safe.</em></p>
<h2 id="limit-from-traditional-bank-system"><a class="toclink" href="#limit-from-traditional-bank-system">Limit from traditional bank system</a></h2>
<p>This contract also has another benefit, a spending limit.
The first version of the contract we developed at Nomadic Labs is called the <strong>Daily Spending Limit</strong>. </p>
<p>This feature is inspired from traditional banking systems.
<strong>Daily limits are placed for security reasons</strong> so that a bank account can’t be drained of an amount of funds in excess of a limit if theft or fraud occurs.</p>
<p><em>Many banks set <span class="caps">ATM</span> withdrawal limits around $500</em></p>
<p>The Daily Spending Limit contract works on a rolling 24h basis.
In the same way as a traditional bank, using the Daily Spending Limit contract, a transaction will be declined if one tries to transfer more than the contract specifies for a 24h window. </p>
<p><strong>Example:</strong> your spending limit is set to 2,000ꜩ per 24 rolling hours. If you transfer 1,500ꜩ at 10.07 am, you can only transfer 500ꜩ until the next day at 10.07 am, at which point the 2,000ꜩ limit is reset.</p>
<p>Unless you amend the contract storage using the corresponding 24-word mnemonic, there is no way to transfer more than the specified spending limit from the contract.
You can also set the daily spending limit to 0.00ꜩ to be sure that no tez will be sent from the <span class="caps">KT1</span> contract.</p>
<h2 id="in-practice"><a class="toclink" href="#in-practice">In practice</a></h2>
<p>You can test this way of using tez via the Cortez wallet for Alphanet<a href="#external-links">*</a></p>
<ol>
<li>Create a new wallet and send it some ꜩ (remember this is the alphanet test network).</li>
<li>Create a Daily Spending Limit Contract and add a few ꜩ from your main account to it.</li>
<li>Go to “Key Management” then remove the master key.</li>
</ol>
<p>You cannot spend tez from your main account (tz1…) anymore because the master key was removed from your phone. However, you can still spend ꜩ from the newly created Daily Spending Limit contract (<span class="caps">KT1</span>…).</p>
<p>If you want to amend the contract’s storage, go to “Key Management” and restore the master key by entering the associated 24-word mnemonic. This way, you will also be able to transfer from your main account (your tz1). <strong>Don’t forget to remove the master key</strong> to maximise security. </p>
<h3 id="i-need-to-make-a-big-transfer"><a class="toclink" href="#i-need-to-make-a-big-transfer">I need to make a big transfer</a></h3>
<p>You can always change your Spending Limit <strong>as long as you have the master key</strong>. In the first alphanet version, users can set their daily spending limits up to 10,000ꜩ per 24h.</p>
<p>This also means that if you do not have the master key encrypted in your phone, or if you do not have access to the 24-word mnemonic, then you cannot update the contract’s storage to spend more than what is indicated in it.</p>
<h3 id="what-if-i-lose-my-phone"><a class="toclink" href="#what-if-i-lose-my-phone">What if I lose my phone?</a></h3>
<p>All you need is a 24-word mnemonic and a new phone.
Download the last version of Cortez, then restore your wallet with the mnemonic. </p>
<p>Once restored, you will be able to amend the storage of your <em>Spending Limit Contract</em> and assign your new secure enclave key to it (the tz3 generated by your new phone).</p>
<p>Once the contract is assigned to your new phone/tz3 address, you can make transfers from it.
Do not forget to remove the master key from your phone in the key management section to maximize security.</p>
<h3 id="peg-leg-pete-stole-my-phone-with-my-password"><a class="toclink" href="#peg-leg-pete-stole-my-phone-with-my-password">Peg-Leg Pete stole my phone with my password.</a></h3>
<p>There are two possible situations:</p>
<ul>
<li>You previously <strong>removed the masterkey</strong> from your phone in Key Management section of the app.</li>
</ul>
<p>Peg-Leg Pete can then only make transfers from your <em>Spending Limit Contract</em>, not your main account (your tz1). He will probably spend up to the specified limit. Without the 24-word mnemonic, and no matter how much he wants to, <strong>Pete cannot steal more than the spending limit in a 24h window</strong>, and not more than the balance of the contract.</p>
<p><img alt="Pete_attack" src="images/slc/pete2.gif"></p>
<p><em>Peg-Leg Pete attacks Mickey © Disney</em></p>
<p>Assuming the contract is a <em>Daily Spending Limit</em> contract, you now have 24h to obtain the 24-word mnemonic, restore the wallet, and assign the Daily Spending Limit contract to the new secure enclave key (tz3…) before Peg-Leg Pete can once again spend the daily spending limit amount.</p>
<ul>
<li><strong>You didn’t remove the masterkey</strong> from your phone.</li>
</ul>
<p>If you can find another phone and the 24-word mnemonic very quickly, you can restore your wallet and amend the contract storage to assign a new tz3. This must be done before Peg-Leg Pete transfers the whole balance to his own wallet.</p>
<h2 id="status-of-the-implementation"><a class="toclink" href="#status-of-the-implementation">Status of the implementation</a></h2>
<p>The first version of the Daily Spending Limit (“<span class="caps">DSL</span>”) smart contract is available on Cortez for Alphanet.
Even if it already works, the feature is still <strong>experimental</strong> and there are some flaws.</p>
<p>The <span class="caps">DSL</span> version <strong>won’t be deployed on mainnet until it is well-tested, fully functional, and formally verified</strong>.
You can find the current version of the <span class="caps">DSL</span> code below, however the code will evolve.</p>
<p>For free Tezzies follow these <a href="https://tezos.gitlab.io/introduction/test_networks.html">instructions</a>. If you have any issues or if you want us to send you Tezzies feel free to ask for help at <a href="mailto:cortez@nomadic-labs.com">cortez@nomadic-labs.com</a>.</p>
<h2 id="formal-verification"><a class="toclink" href="#formal-verification">Formal verification</a></h2>
<p>You can find a link below to the very first version of the <span class="caps">DSL</span>, currently implemented in the Android version of Cortez for alphanet.</p>
<p>The contract has been partially verified using <a href="https://gitlab.com/nomadic-labs/mi-cho-coq/">Mi-Cho-Coq</a>, a specification of Michelson in Coq to prove properties about smart contracts in Tezos. There is more work to be done.</p>
<p>As stated above, the contract is meant to evolve and improve with subsequent versions until it is sufficiently flawless before it will be deployed on mainnet.</p>
<h2 id="michelson-code"><a class="toclink" href="#michelson-code">Michelson code</a></h2>
<p>This is the current version of the Daily Spending Limit (<span class="caps">DSL</span>) contract code.
Note that this version will change over time.</p>
<div class="highlight"><pre><span></span><code><span class="k">storage</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span>
<span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">%salt</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">key_hash</span><span class="w"> </span><span class="k k-Name">%daily_key_hash</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span>
<span class="w"> </span><span class="p">(</span><span class="kt">mutez</span><span class="w"> </span><span class="k k-Name">%remaining_funds</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="k k-Name">%blocking_period</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="k k-Name">%file</span>
<span class="w"> </span><span class="p">(</span><span class="kt">list</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="kt">timestamp</span><span class="w"> </span><span class="kt">mutez</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">list</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="kt">timestamp</span><span class="w"> </span><span class="kt">mutez</span><span class="p">)))))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span>
<span class="w"> </span><span class="p">(</span><span class="kt">key_hash</span><span class="w"> </span><span class="k k-Name">%master_key_hash</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">%salt</span><span class="p">)));</span>
<span class="k">parameter</span><span class="w"> </span><span class="p">(</span><span class="kt">option</span>
<span class="w"> </span><span class="p">(</span><span class="kt">or</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="k k-Name">%contract_renewal</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span>
<span class="w"> </span><span class="p">(</span><span class="kt">key</span><span class="w"> </span><span class="k k-Name">%public_key</span><span class="p">)</span>
<span class="w"> </span><span class="kt">signature</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span>
<span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">%salt</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">key_hash</span><span class="w"> </span><span class="k k-Name">%daily_key_hash</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span>
<span class="w"> </span><span class="p">(</span><span class="kt">mutez</span><span class="w"> </span><span class="k k-Name">%remaining_funds</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">int</span><span class="w"> </span><span class="k k-Name">%blocking_period</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="k k-Name">%file</span>
<span class="w"> </span><span class="p">(</span><span class="kt">list</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="kt">timestamp</span><span class="w"> </span><span class="kt">mutez</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">list</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="kt">timestamp</span><span class="w"> </span><span class="kt">mutez</span><span class="p">)))))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">key_hash</span><span class="w"> </span><span class="k k-Name">%new_master_key_hash</span><span class="p">)))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="k k-Name">%transfer</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span>
<span class="w"> </span><span class="p">(</span><span class="kt">list</span><span class="w"> </span><span class="k k-Name">%beneficiaries</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span>
<span class="w"> </span><span class="p">(</span><span class="kt">mutez</span><span class="w"> </span><span class="k k-Name">%amount</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">contract</span><span class="w"> </span><span class="k k-Name">%beneficiary</span><span class="w"> </span><span class="kt">unit</span><span class="p">)))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">key_hash</span><span class="w"> </span><span class="k k-Name">%new_daily_key_hash</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span>
<span class="w"> </span><span class="p">(</span><span class="kt">key</span><span class="w"> </span><span class="k k-Name">%public_key</span><span class="p">)</span>
<span class="w"> </span><span class="kt">signature</span><span class="p">))));</span>
<span class="k">code</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">UNPAPAIR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">IF_SOME</span><span class="p">{</span><span class="w"> </span><span class="kr">IF_LEFT</span><span class="p">{</span><span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="kr">DROP</span><span class="p">};</span><span class="w"> </span><span class="c1"># call by master key</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">UNPAIR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">UNPAIR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">PACK</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">nat</span><span class="w"> </span><span class="m">1</span><span class="p">;</span>
<span class="w"> </span><span class="kr">ADD</span><span class="p">;</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">PACK</span><span class="p">};</span>
<span class="w"> </span><span class="kr">SWAP</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="kr">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DUP</span><span class="p">};</span>
<span class="w"> </span><span class="kr">UNPAIR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">HASH_KEY</span><span class="p">;</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="kr">ASSERT_CMPEQ</span><span class="p">;</span>
<span class="w"> </span><span class="kr">CHECK_SIGNATURE</span><span class="p">;</span>
<span class="w"> </span><span class="kr">IF</span><span class="p">{</span><span class="kr">DROP</span><span class="p">}{</span><span class="kr">FAILWITH</span><span class="p">};</span>
<span class="w"> </span><span class="kr">UNPAIR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="kr">PAIR</span><span class="p">};</span>
<span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="kt">operation</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">DIP</span><span class="w"> </span><span class="p">{</span><span class="kr">UNPAPAIR</span><span class="p">};</span><span class="w"> </span><span class="c1"># transaction call</span>
<span class="w"> </span><span class="kr">UNPAIR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">PACK</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">UNPAIR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="kr">SWAP</span><span class="p">}};</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="kr">SWAP</span><span class="p">};</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">UNPAIR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">HASH_KEY</span><span class="p">};</span>
<span class="w"> </span><span class="kr">UNPAIR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">ASSERT_CMPEQ</span><span class="p">};</span><span class="w"> </span><span class="c1"># keys compatibility verification</span>
<span class="w"> </span><span class="kr">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">nat</span><span class="w"> </span><span class="m">1</span><span class="p">;</span>
<span class="w"> </span><span class="kr">ADD</span><span class="p">;</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">PACK</span><span class="p">;</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="kr">CONCAT</span><span class="p">;</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="kr">DUP</span><span class="p">}};</span>
<span class="w"> </span><span class="kr">CHECK_SIGNATURE</span><span class="p">;</span><span class="w"> </span><span class="c1"># signature verification</span>
<span class="w"> </span><span class="kr">IF</span><span class="p">{</span><span class="kr">DROP</span><span class="p">}{</span><span class="kr">FAILWITH</span><span class="p">}};</span>
<span class="w"> </span><span class="kr">PAIR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">UNPAIR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">UNPAIR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="kc">True</span><span class="p">;</span>
<span class="w"> </span><span class="kr">LOOP</span><span class="p">{</span><span class="w"> </span><span class="c1"># search for funds to unlock</span>
<span class="w"> </span><span class="kr">IF_CONS</span><span class="p">{</span><span class="w"> </span><span class="kr">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">CAR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">NOW</span><span class="p">;</span>
<span class="w"> </span><span class="kr">IFCMPGE</span><span class="p">{</span><span class="w"> </span><span class="kr">CDR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">ADD</span><span class="p">}};</span>
<span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="kc">True</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">CONS</span><span class="p">;</span>
<span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="kc">False</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">IF_CONS</span><span class="p">{</span><span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="kt">timestamp</span><span class="w"> </span><span class="kt">mutez</span><span class="p">);</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">CONS</span><span class="p">;</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">ITER</span><span class="p">{</span><span class="w"> </span><span class="kr">CONS</span><span class="p">};</span>
<span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="kt">timestamp</span><span class="w"> </span><span class="kt">mutez</span><span class="p">);</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="kc">True</span><span class="p">}</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="kt">timestamp</span><span class="w"> </span><span class="kt">mutez</span><span class="p">);</span>
<span class="w"> </span><span class="kr">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">bool</span><span class="w"> </span><span class="kc">False</span><span class="p">}</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">SWAP</span><span class="p">};</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">mutez</span><span class="w"> </span><span class="m">0</span><span class="p">;</span>
<span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="kt">operation</span><span class="p">;</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="kr">ITER</span><span class="p">{</span><span class="w"> </span><span class="kr">UNPAIR</span><span class="p">;</span><span class="w"> </span><span class="c1"># transactions production</span>
<span class="w"> </span><span class="kr">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span>
<span class="w"> </span><span class="kr">UNIT</span><span class="p">;</span>
<span class="w"> </span><span class="kr">TRANSFER_TOKENS</span><span class="p">;</span>
<span class="w"> </span><span class="kr">CONS</span><span class="p">;</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">ADD</span><span class="p">}</span><span class="w"> </span><span class="c1"># blocking period computation</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span><span class="w"> </span><span class="c1"># spent funds lock</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">NOW</span><span class="p">;</span>
<span class="w"> </span><span class="kr">ADD</span><span class="p">;</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="kr">SWAP</span><span class="p">};</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DUP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">PAIR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="p">{</span><span class="w"> </span><span class="kr">CONS</span><span class="p">};</span><span class="w"> </span><span class="c1"># add to the queue</span>
<span class="w"> </span><span class="kr">PAIR</span><span class="p">}};</span>
<span class="w"> </span><span class="kr">SUB</span><span class="p">};</span><span class="w"> </span><span class="c1"># unlocked remaining funds computation</span>
<span class="w"> </span><span class="kr">SWAP</span><span class="p">;</span>
<span class="w"> </span><span class="kr">PAIR</span><span class="p">;</span>
<span class="w"> </span><span class="kr">PAIR</span><span class="p">;</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="kr">PAIR</span><span class="p">};</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="kt">operation</span><span class="p">;</span><span class="w"> </span><span class="c1"># simple fund addition (parameter: None)</span>
<span class="w"> </span><span class="p">};</span>
<span class="w"> </span><span class="kr">PAPAIR</span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
</code></pre></div>
<h2 id="external-links"><a class="toclink" href="#external-links"><a id="external-links"></a>External links</a></h2>
<ul>
<li>
<p><a href="https://en.wikipedia.org/wiki/KISS_principle"><span class="caps">KISS</span> Principle</a></p>
</li>
<li>
<p><a href="https://play.google.com/store/apps/details?id=com.tezcore.cortez.alphanet">Google Play</a>.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/cortez-android">Source code</a></p>
</li>
</ul>Babylon update instructions for delegation wallet developers2019-08-27T17:43:00+02:002019-08-27T17:43:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2019-08-27:/babylon-update-instructions-for-delegation-wallet-developers.html<p>How to update the delegation feature of your wallet for 005 (aka. Babylon)</p><h2 id="introduction"><a class="toclink" href="#introduction">Introduction</a></h2>
<p>Tezos wallets usually feature management of scriptless originated
(aka <span class="caps">KT1</span>) accounts used to delegate tokens.</p>
<p>This document details the steps needed for wallet developers to update
their applications in anticipation to the breaking changes in the
Babylon protocol update. See also
<a href="https://gitlab.com/metastatedev/tezos/-/blob/96c655929496b2e59dbf965893fb378e2a6e4a08/specs/migration_004_to_005.md">cryptium’s migration guidelines</a>
and <a href="https://tezos.gitlab.io/protocols/005_babylon.html">Babylon’s documentation</a> for
more technical details on the Babylon update.</p>
<p>The Babylon protocol update brings two big changes to the way
delegation can be implemented. First, implicit (aka tz) accounts can
now directly delegate their tokens (see
relevant <a href="https://tezos.gitlab.io/protocols/005_babylon.html#make-implicit-accounts-delegatable">documentation’s section</a>). Second,
the scriptless <span class="caps">KT1</span> accounts, whose main purpose in Athens were
delegation, are replaced by smart contracts whose address is unchanged
and script is
<a href="https://gitlab.com/nomadic-labs/mi-cho-coq/blob/master/src/contracts/manager.tz">manager.tz</a>.</p>
<p>The “delegation” operation, which was restricted to <span class="caps">KT1</span> accounts in
Athens, is now restricted to tz implicit accounts in Babylon. To change or
withdraw the delegate of a smart contract in Babylon, the
“SET_DELEGATE” Michelson instruction has to be used.</p>
<p>Moreover, smart contract in Babylon cannot be the source of a
transaction signed by their manager; the source of a transaction is
always a tz implicit account. To spend the tokens of a smart contract
running the manager.tz script, its manager sends a 0tz transaction to
the smart contract with an argument describing the operation that the
smart contract should perform.</p>
<p>Concretely, this means that wallets can be updated in one of the following ways:</p>
<ol>
<li>
<p>Remove support for <span class="caps">KT1</span> delegation accounts (pros: simpler, more
future proof and aligned with the new accounts taxonomy).<br>
Since tz accounts in Babylon can be delegated, <span class="caps">KT1</span> accounts are not
needed anymore for a wallet to feature delegation.<br>
The Babylon’s “delegation” operation on tz accounts is very similar
to the Athens’ “delegation” operation on <span class="caps">KT1</span> accounts so updating
the wallet should be relatively easy in this case. However,
migrating existing accounts would be needed. To learn how to transfer the tokens
of a <span class="caps">KT1</span> account to an implicit account (its manager for example), see <a href="#transfer-from-a-managertz-smart-contract-to-an-implicit-tz-account">next section</a>.</p>
</li>
<li>
<p>Interact with the manager.tz script (pro: user addresses do not
change)<br>
The second option is to keep all tokens and delegations as they are
and adapt the wallet code to interact with the manager.tz smart
contract script that all currently scriptless <span class="caps">KT1</span> accounts will run
after the Babylon migration.<br>
In this case all operations related to <span class="caps">KT1</span> acconts need to be
adapted but no account migration is needed. The following sections
detail the changes in the operations.</p>
</li>
</ol>
<h2 id="transfer-from-a-managertz-smart-contract-to-an-implicit-tz-account"><a class="toclink" href="#transfer-from-a-managertz-smart-contract-to-an-implicit-tz-account">Transfer from a manager.tz smart contract to an implicit (tz) account</a></h2>
<p>To transfer <code><amount></code> tokens from a manager.tz smart contract to a tz
account whose key hash is <code><destination></code>, we need to call the smart
contract on its “%do” entrypoint with a lambda that builds the desired
transfer as argument. The
<a href="https://gitlab.com/metastatedev/tezos/-/blob/96c655929496b2e59dbf965893fb378e2a6e4a08/specs/migration_004_to_005.md#managertz-script">migration instructions from Cryptium Labs</a>
contain the required Michelson code:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"> </span><span class="kr">DROP</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="kt">operation</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">key_hash</span><span class="w"> </span><span class="err"><</span><span class="kt">destination</span><span class="err">></span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">IMPLICIT_ACCOUNT</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">mutez</span><span class="w"> </span><span class="err"><</span><span class="kt">amount</span><span class="err">></span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">UNIT</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">TRANSFER_TOKENS</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">CONS</span><span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>The fees, gas limit, and storage limit for this call can be set to the
following values:</p>
<ul>
<li>
<p>fees: 2941 μꜩ</p>
</li>
<li>
<p>gas limit: 26283</p>
</li>
<li>
<p>storage limit: 0 byte</p>
</li>
</ul>
<p>The amount of the transaction to the smart contract must be 0ꜩ.</p>
<p>Before calling the smart contract, we need to compute and sign the
binary representation of the transfer operation. To get a description
of the binary format, we use a Babylon command line client as follows:
<code>tezos-client describe unsigned operation</code>. To only get what has
changed in Babylon, we can also read
<a href="https://tezos.gitlab.io/protocols/005_babylon.html#changes-to-the-binary-format-of-operations">the online documentation</a>.</p>
<p>The byte sequence of the operation we want can be decomposed as follows:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="o"><</span><span class="mi">32</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="nv">block</span><span class="w"> </span><span class="nv">hash</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">current</span><span class="w"> </span><span class="nv">head</span><span class="w"> </span><span class="nv">block</span><span class="w"> </span>[<span class="ss">(</span><span class="nv">see</span><span class="w"> </span><span class="nv">below</span><span class="ss">)</span>]<span class="ss">(</span>#<span class="nv">getting</span><span class="o">-</span><span class="nv">information</span><span class="o">-</span><span class="nv">to</span><span class="o">-</span><span class="nv">craft</span><span class="o">-</span><span class="nv">the</span><span class="o">-</span><span class="nv">operations</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x6c</span>:<span class="w"> </span><span class="nv">Transaction</span><span class="w"> </span><span class="nv">tag</span><span class="w"> </span><span class="ss">(</span><span class="mi">108</span><span class="ss">)</span>
<span class="w"> </span><span class="o"><</span><span class="mi">21</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="nv">key</span><span class="w"> </span><span class="nv">hash</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">source</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">transaction</span><span class="w"> </span><span class="ss">(</span><span class="nv">must</span><span class="w"> </span><span class="nv">be</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">manager</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">smart</span><span class="w"> </span><span class="nv">contract</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">xfd16</span>:<span class="w"> </span><span class="nv">fees</span><span class="w"> </span><span class="ss">(</span><span class="mi">2941</span><span class="w"> </span>μꜩ<span class="ss">)</span>
<span class="w"> </span><span class="o"><</span><span class="nv">counter</span><span class="o">></span>:<span class="w"> </span><span class="nv">counter</span><span class="w"> </span><span class="nv">associated</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">manager</span><span class="w"> </span><span class="nv">tz</span><span class="w"> </span><span class="nv">account</span><span class="w"> </span>[<span class="ss">(</span><span class="nv">see</span><span class="w"> </span><span class="nv">below</span><span class="ss">)</span>]<span class="ss">(</span>#<span class="nv">getting</span><span class="o">-</span><span class="nv">information</span><span class="o">-</span><span class="nv">to</span><span class="o">-</span><span class="nv">craft</span><span class="o">-</span><span class="nv">the</span><span class="o">-</span><span class="nv">operations</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">xabcd01</span>:<span class="w"> </span><span class="nv">gas_limit</span><span class="w"> </span><span class="ss">(</span><span class="mi">26283</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x00</span>:<span class="w"> </span><span class="nv">storage_limit</span><span class="w"> </span><span class="ss">(</span><span class="mi">0</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x00</span>:<span class="w"> </span><span class="nv">amount</span><span class="w"> </span><span class="ss">(</span><span class="mi">0</span>ꜩ<span class="ss">)</span>
<span class="w"> </span><span class="o"><</span><span class="mi">22</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">address</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">smart</span><span class="w"> </span><span class="nv">contract</span><span class="w"> </span><span class="ss">(</span><span class="nv">must</span><span class="w"> </span><span class="nv">start</span><span class="w"> </span><span class="nv">with</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">byte</span><span class="w"> </span><span class="mi">0</span><span class="nv">x01</span><span class="w"> </span><span class="nv">and</span><span class="w"> </span><span class="k">end</span><span class="w"> </span><span class="nv">with</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">byte</span><span class="w"> </span><span class="mi">0</span><span class="nv">x00</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">xff</span><span class="w"> </span><span class="ss">(</span><span class="nv">or</span><span class="w"> </span><span class="nv">any</span><span class="w"> </span><span class="nv">other</span><span class="w"> </span><span class="nv">non</span><span class="o">-</span><span class="nv">null</span><span class="w"> </span><span class="nv">byte</span><span class="ss">)</span>:<span class="w"> </span><span class="nv">presence</span><span class="w"> </span><span class="nv">flag</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">parameters</span><span class="w"> </span><span class="ss">(</span><span class="nv">entrypoint</span><span class="w"> </span><span class="nv">and</span><span class="w"> </span><span class="nv">argument</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x02</span>:<span class="w"> </span><span class="nv">tag</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="s2">"%do"</span><span class="w"> </span><span class="nv">entrypoint</span>
<span class="w"> </span><span class="o"><</span><span class="mi">4</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="nv">length</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">argument</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x02</span>:<span class="w"> </span><span class="nv">Michelson</span><span class="w"> </span><span class="nv">sequence</span>
<span class="w"> </span><span class="o"><</span><span class="mi">4</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="nv">length</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">sequence</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x0320</span>:<span class="w"> </span><span class="nv">DROP</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x053d</span>:<span class="w"> </span><span class="nv">NIL</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x036d</span>:<span class="w"> </span><span class="nv">operation</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x0743</span>:<span class="w"> </span><span class="nv">PUSH</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x035d</span>:<span class="w"> </span><span class="nv">key_hash</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x0a</span>:<span class="w"> </span><span class="nv">Byte</span><span class="w"> </span><span class="nv">sequence</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x00000015</span>:<span class="w"> </span><span class="nv">Length</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">sequence</span><span class="w"> </span><span class="ss">(</span><span class="mi">21</span><span class="w"> </span><span class="nv">bytes</span><span class="ss">)</span>
<span class="w"> </span><span class="o"><</span><span class="mi">21</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="o"><</span><span class="nv">destination</span><span class="o">></span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x031e</span>:<span class="w"> </span><span class="nv">IMPLICIT_ACCOUNT</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x0743</span>:<span class="w"> </span><span class="nv">PUSH</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x036a</span>:<span class="w"> </span><span class="nv">mutez</span>
<span class="w"> </span><span class="o"><</span><span class="nv">amount</span><span class="o">></span>:<span class="w"> </span><span class="nv">Amout</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">be</span><span class="w"> </span><span class="nv">transfered</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x034f</span>:<span class="w"> </span><span class="nv">UNIT</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x034d</span>:<span class="w"> </span><span class="nv">TRANSFER_TOKENS</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x031b</span>:<span class="w"> </span><span class="nv">CONS</span>
</code></pre></div>
<p>Once signed, this operation can be sent to the node by invoking the following <span class="caps">RPC</span>:</p>
<div class="highlight"><pre><span></span><code><span class="o">/</span><span class="nv">chains</span><span class="o">/</span><span class="nv">main</span><span class="o">/</span><span class="nv">blocks</span><span class="o">/</span><span class="nv">head</span><span class="o">/</span><span class="nv">helpers</span><span class="o">/</span><span class="nv">preapply</span><span class="o">/</span><span class="nv">operations</span>
<span class="w"> </span>[<span class="w"> </span>{<span class="w"> </span><span class="s2">"protocol"</span>:<span class="w"> </span><span class="s2">"PsBABY5HQTSkA4297zNHfsZNKtxULfL18y95qb3m53QJiXGmrbU"</span>,
<span class="w"> </span><span class="s2">"branch"</span>:<span class="w"> </span><span class="s2">"<block hash of the head block in Base58 [(see below)](#getting-information-to-craft-the-operations)>"</span>,
<span class="w"> </span><span class="s2">"contents"</span>:
<span class="w"> </span>[<span class="w"> </span>{<span class="w"> </span><span class="s2">"kind"</span>:<span class="w"> </span><span class="s2">"transaction"</span>,
<span class="w"> </span><span class="s2">"source"</span>:<span class="w"> </span><span class="s2">"<manager key hash>"</span>,<span class="w"> </span><span class="s2">"fee"</span>:<span class="w"> </span><span class="s2">"2941"</span>,
<span class="w"> </span><span class="s2">"counter"</span>:<span class="w"> </span><span class="s2">"<counter of the manager [(see below)](#getting-information-to-craft-the-operations)>"</span>,<span class="w"> </span><span class="s2">"gas_limit"</span>:<span class="w"> </span><span class="s2">"26283"</span>,<span class="w"> </span><span class="s2">"storage_limit"</span>:<span class="w"> </span><span class="s2">"0"</span>,
<span class="w"> </span><span class="s2">"amount"</span>:<span class="w"> </span><span class="s2">"0"</span>,
<span class="w"> </span><span class="s2">"destination"</span>:<span class="w"> </span><span class="s2">"<address of the smart contract in Base58 (must start with "</span><span class="nv">KT1</span><span class="s2">")>"</span>,
<span class="w"> </span><span class="s2">"parameters"</span>:
<span class="w"> </span>{<span class="w"> </span><span class="s2">"entrypoint"</span>:<span class="w"> </span><span class="s2">"do"</span>,
<span class="w"> </span><span class="s2">"value"</span>:
<span class="w"> </span>[<span class="w"> </span>{<span class="w"> </span><span class="s2">"prim"</span>:<span class="w"> </span><span class="s2">"DROP"</span><span class="w"> </span>},
<span class="w"> </span>{<span class="w"> </span><span class="s2">"prim"</span>:<span class="w"> </span><span class="s2">"NIL"</span>,<span class="w"> </span><span class="s2">"args"</span>:<span class="w"> </span>[<span class="w"> </span>{<span class="w"> </span><span class="s2">"prim"</span>:<span class="w"> </span><span class="s2">"operation"</span><span class="w"> </span>}<span class="w"> </span>]<span class="w"> </span>},
<span class="w"> </span>{<span class="w"> </span><span class="s2">"prim"</span>:<span class="w"> </span><span class="s2">"PUSH"</span>,
<span class="w"> </span><span class="s2">"args"</span>:
<span class="w"> </span>[<span class="w"> </span>{<span class="w"> </span><span class="s2">"prim"</span>:<span class="w"> </span><span class="s2">"key_hash"</span><span class="w"> </span>},
<span class="w"> </span>{<span class="w"> </span><span class="s2">"bytes"</span>:
<span class="w"> </span><span class="s2">"<destination (in hexadecimal without the leading '0x')>"</span><span class="w"> </span>}<span class="w"> </span>]<span class="w"> </span>},
<span class="w"> </span>{<span class="w"> </span><span class="s2">"prim"</span>:<span class="w"> </span><span class="s2">"IMPLICIT_ACCOUNT"</span><span class="w"> </span>},
<span class="w"> </span>{<span class="w"> </span><span class="s2">"prim"</span>:<span class="w"> </span><span class="s2">"PUSH"</span>,
<span class="w"> </span><span class="s2">"args"</span>:
<span class="w"> </span>[<span class="w"> </span>{<span class="w"> </span><span class="s2">"prim"</span>:<span class="w"> </span><span class="s2">"mutez"</span><span class="w"> </span>},<span class="w"> </span>{<span class="w"> </span><span class="s2">"int"</span>:<span class="w"> </span><span class="s2">"<amount>"</span><span class="w"> </span>}<span class="w"> </span>]<span class="w"> </span>},
<span class="w"> </span>{<span class="w"> </span><span class="s2">"prim"</span>:<span class="w"> </span><span class="s2">"UNIT"</span><span class="w"> </span>},<span class="w"> </span>{<span class="w"> </span><span class="s2">"prim"</span>:<span class="w"> </span><span class="s2">"TRANSFER_TOKENS"</span><span class="w"> </span>},
<span class="w"> </span>{<span class="w"> </span><span class="s2">"prim"</span>:<span class="w"> </span><span class="s2">"CONS"</span><span class="w"> </span>}<span class="w"> </span>]<span class="w"> </span>}<span class="w"> </span>}<span class="w"> </span>],
<span class="w"> </span><span class="s2">"signature"</span>:
<span class="w"> </span><span class="s2">"<signature>"</span><span class="w"> </span>}<span class="w"> </span>]
</code></pre></div>
<h2 id="transfer-from-a-managertz-smart-contract-to-another-smart-contract"><a class="toclink" href="#transfer-from-a-managertz-smart-contract-to-another-smart-contract">Transfer from a manager.tz smart contract to another smart contract</a></h2>
<p>Transferring to a smart contract is very similar. The main difference
is that we need to check the smart contract type in Michelson using
the expensive <code>CONTRACT</code> instruction.</p>
<p>The lambda to pass as parameter to the manager.tz smart contract to
make it send <code><amount></code> tokens to the smart contract at address
<code><destination></code>, calling it on the entrypoint <code><entrypoint></code> of type
<code><ty></code> with the parameter <code><param></code> is again given in the <a href="https://gitlab.com/metastatedev/tezos/-/blob/96c655929496b2e59dbf965893fb378e2a6e4a08/specs/migration_004_to_005.md#managertz-script">migration
instructions from Cryptium
Labs</a>:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"> </span><span class="kr">DROP</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="kt">operation</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">address</span><span class="w"> </span><span class="err"><</span><span class="kt">destination</span><span class="err">></span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">CONTRACT</span><span class="w"> </span><span class="k k-Name">%</span><span class="err"><</span><span class="kt">entrypoint</span><span class="err">></span><span class="w"> </span><span class="err"><</span><span class="kt">ty</span><span class="err">></span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">ASSERT_SOME</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">mutez</span><span class="w"> </span><span class="err"><</span><span class="kt">amout</span><span class="err">></span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="err"><</span><span class="kt">ty</span><span class="err">></span><span class="w"> </span><span class="err"><</span><span class="kt">param</span><span class="err">></span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">TRANSFER_TOKENS</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">CONS</span><span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>Note that the entrypoint must be omitted if it is <code>default</code> (that is,
you should write <code>CONTRACT <ty></code> instead of <code>CONTRACT %default <ty></code>).</p>
<p>If <code><ty></code> is <code>unit</code>, the instruction <code>PUSH unit <param></code> can be
replaced by the slightly cheaper <code>UNIT</code> instruction.</p>
<p>The fees and gas limit will depend on the size of the
parameter. In the particular case of a transfer to another manager.tz smart
contract, the gas limit should be set to <code>44725</code>.</p>
<p>Storage limit can still be set to 0 bytes.</p>
<p>The amount of the transaction to the smart contract must again be 0ꜩ.</p>
<p>The byte sequence of the operation we want can be decomposed as follows:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="o"><</span><span class="nt">32</span><span class="w"> </span><span class="nt">bytes</span><span class="o">>:</span><span class="w"> </span><span class="nt">block</span><span class="w"> </span><span class="nt">hash</span><span class="w"> </span><span class="nt">of</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">current</span><span class="w"> </span><span class="nt">head</span><span class="w"> </span><span class="nt">block</span><span class="w"> </span><span class="cp">[</span><span class="p">(</span><span class="nx">see</span><span class="w"> </span><span class="nx">below</span><span class="p">)</span><span class="cp">]</span><span class="o">(</span><span class="p">#</span><span class="nn">getting-information-to-craft-the-operations</span><span class="o">)</span>
<span class="w"> </span><span class="nt">0x6c</span><span class="o">:</span><span class="w"> </span><span class="nt">Transaction</span><span class="w"> </span><span class="nt">tag</span><span class="w"> </span><span class="o">(</span><span class="nt">108</span><span class="o">)</span>
<span class="w"> </span><span class="o"><</span><span class="nt">21</span><span class="w"> </span><span class="nt">bytes</span><span class="o">>:</span><span class="w"> </span><span class="nt">key</span><span class="w"> </span><span class="nt">hash</span><span class="w"> </span><span class="nt">of</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">source</span><span class="w"> </span><span class="nt">of</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">transaction</span><span class="w"> </span><span class="o">(</span><span class="nt">must</span><span class="w"> </span><span class="nt">be</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">manager</span><span class="w"> </span><span class="nt">of</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">smart</span><span class="w"> </span><span class="nt">contract</span><span class="o">)</span>
<span class="w"> </span><span class="o"><</span><span class="nt">fees</span><span class="o">></span>
<span class="w"> </span><span class="o"><</span><span class="nt">counter</span><span class="o">>:</span><span class="w"> </span><span class="nt">counter</span><span class="w"> </span><span class="nt">associated</span><span class="w"> </span><span class="nt">to</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">manager</span><span class="w"> </span><span class="nt">tz</span><span class="w"> </span><span class="nt">account</span><span class="w"> </span><span class="cp">[</span><span class="p">(</span><span class="nx">see</span><span class="w"> </span><span class="nx">below</span><span class="p">)</span><span class="cp">]</span><span class="o">(</span><span class="p">#</span><span class="nn">getting-information-to-craft-the-operations</span><span class="o">)</span>
<span class="w"> </span><span class="o"><</span><span class="nt">gas_limit</span><span class="o">></span>
<span class="w"> </span><span class="nt">0x00</span><span class="o">:</span><span class="w"> </span><span class="nt">storage_limit</span><span class="w"> </span><span class="o">(</span><span class="nt">0</span><span class="o">)</span>
<span class="w"> </span><span class="nt">0x00</span><span class="o">:</span><span class="w"> </span><span class="nt">amount</span><span class="w"> </span><span class="o">(</span><span class="nt">0ꜩ</span><span class="o">)</span>
<span class="w"> </span><span class="o"><</span><span class="nt">22</span><span class="w"> </span><span class="nt">bytes</span><span class="o">>:</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">address</span><span class="w"> </span><span class="nt">of</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">manager</span><span class="p">.</span><span class="nc">tz</span><span class="w"> </span><span class="nt">smart</span><span class="w"> </span><span class="nt">contract</span><span class="w"> </span><span class="o">(</span><span class="nt">must</span><span class="w"> </span><span class="nt">start</span><span class="w"> </span><span class="nt">with</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">byte</span><span class="w"> </span><span class="nt">0x01</span><span class="w"> </span><span class="nt">and</span><span class="w"> </span><span class="nt">end</span><span class="w"> </span><span class="nt">with</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">byte</span><span class="w"> </span><span class="nt">0x00</span><span class="o">)</span>
<span class="w"> </span><span class="nt">0xff</span><span class="w"> </span><span class="o">(</span><span class="nt">or</span><span class="w"> </span><span class="nt">any</span><span class="w"> </span><span class="nt">other</span><span class="w"> </span><span class="nt">non-null</span><span class="w"> </span><span class="nt">byte</span><span class="o">):</span><span class="w"> </span><span class="nt">presence</span><span class="w"> </span><span class="nt">flag</span><span class="w"> </span><span class="nt">for</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">parameters</span><span class="w"> </span><span class="o">(</span><span class="nt">entrypoint</span><span class="w"> </span><span class="nt">and</span><span class="w"> </span><span class="nt">argument</span><span class="o">)</span>
<span class="w"> </span><span class="nt">0x02</span><span class="o">:</span><span class="w"> </span><span class="nt">tag</span><span class="w"> </span><span class="nt">of</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="s2">"%do"</span><span class="w"> </span><span class="nt">entrypoint</span>
<span class="w"> </span><span class="o"><</span><span class="nt">4</span><span class="w"> </span><span class="nt">bytes</span><span class="o">>:</span><span class="w"> </span><span class="nt">length</span><span class="w"> </span><span class="nt">of</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">argument</span>
<span class="w"> </span><span class="nt">0x02</span><span class="o">:</span><span class="w"> </span><span class="nt">Michelson</span><span class="w"> </span><span class="nt">sequence</span>
<span class="w"> </span><span class="o"><</span><span class="nt">4</span><span class="w"> </span><span class="nt">bytes</span><span class="o">>:</span><span class="w"> </span><span class="nt">length</span><span class="w"> </span><span class="nt">of</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">sequence</span>
<span class="w"> </span><span class="nt">0x0320</span><span class="o">:</span><span class="w"> </span><span class="nt">DROP</span>
<span class="w"> </span><span class="nt">0x053d</span><span class="o">:</span><span class="w"> </span><span class="nt">NIL</span>
<span class="w"> </span><span class="nt">0x036d</span><span class="o">:</span><span class="w"> </span><span class="nt">operation</span>
<span class="w"> </span><span class="nt">0x0743</span><span class="o">:</span><span class="w"> </span><span class="nt">PUSH</span>
<span class="w"> </span><span class="nt">0x036e</span><span class="o">:</span><span class="w"> </span><span class="nt">address</span>
<span class="w"> </span><span class="nt">0x0a</span><span class="o">:</span><span class="w"> </span><span class="nt">byte</span><span class="w"> </span><span class="nt">sequence</span>
<span class="w"> </span><span class="nt">0x00000016</span><span class="o">:</span><span class="w"> </span><span class="nt">Length</span><span class="w"> </span><span class="nt">of</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">sequence</span><span class="w"> </span><span class="o">(</span><span class="nt">22</span><span class="w"> </span><span class="nt">bytes</span><span class="o">)</span>
<span class="w"> </span><span class="o"><</span><span class="nt">22</span><span class="w"> </span><span class="nt">bytes</span><span class="o">>:</span><span class="w"> </span><span class="o"><</span><span class="nt">destination</span><span class="o">></span><span class="w"> </span><span class="o">(</span><span class="nt">must</span><span class="w"> </span><span class="nt">start</span><span class="w"> </span><span class="nt">with</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">byte</span><span class="w"> </span><span class="nt">0x01</span><span class="w"> </span><span class="nt">and</span><span class="w"> </span><span class="nt">end</span><span class="w"> </span><span class="nt">with</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">byte</span><span class="w"> </span><span class="nt">0x00</span><span class="o">)</span>
<span class="w"> </span><span class="o">(</span><span class="nt">if</span><span class="w"> </span><span class="nt">the</span><span class="w"> </span><span class="nt">entrypoint</span><span class="w"> </span><span class="nt">is</span><span class="w"> </span><span class="s2">"default"</span>
<span class="w"> </span><span class="nt">0x0555</span><span class="o">:</span><span class="w"> </span><span class="nt">CONTRACT</span>
<span class="w"> </span><span class="o"><</span><span class="nt">ty</span><span class="o">></span>
<span class="w"> </span><span class="nt">else</span>
<span class="w"> </span><span class="nt">0x0655</span><span class="o">:</span><span class="w"> </span><span class="nt">CONTRACT</span>
<span class="w"> </span><span class="o"><</span><span class="nt">ty</span><span class="o">></span>
<span class="w"> </span><span class="o"><</span><span class="nt">entrypoint</span><span class="o">></span>
<span class="w"> </span><span class="o">)</span>
<span class="w"> </span><span class="nt">0x0200000015072f02000000090200000004034f03270200000000</span><span class="o">:</span><span class="w"> </span><span class="nt">ASSERT_SOME</span><span class="w"> </span><span class="o">(</span><span class="nt">unfolded</span><span class="w"> </span><span class="nt">as</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">IF_NONE</span><span class="w"> </span><span class="err">{</span><span class="w"> </span><span class="err">{</span><span class="w"> </span><span class="err">UNIT</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="err">FAILWITH</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="err">}</span><span class="w"> </span><span class="p">{}</span><span class="w"> </span><span class="err">}</span><span class="o">)</span>
<span class="w"> </span><span class="nt">0x0743</span><span class="o">:</span><span class="w"> </span><span class="nt">PUSH</span>
<span class="w"> </span><span class="nt">0x036a</span><span class="o">:</span><span class="w"> </span><span class="nt">mutez</span>
<span class="w"> </span><span class="o"><</span><span class="nt">amount</span><span class="o">></span>
<span class="w"> </span><span class="o">(</span><span class="nt">if</span><span class="w"> </span><span class="o"><</span><span class="nt">ty</span><span class="o">></span><span class="w"> </span><span class="nt">is</span><span class="w"> </span><span class="nt">unit</span>
<span class="w"> </span><span class="nt">0x4f</span><span class="o">:</span><span class="w"> </span><span class="nt">UNIT</span>
<span class="w"> </span><span class="nt">else</span>
<span class="w"> </span><span class="nt">0x0743</span><span class="o">:</span><span class="w"> </span><span class="nt">PUSH</span>
<span class="w"> </span><span class="o"><</span><span class="nt">ty</span><span class="o">></span>
<span class="w"> </span><span class="o"><</span><span class="nt">param</span><span class="o">></span>
<span class="w"> </span><span class="o">)</span>
<span class="w"> </span><span class="nt">0x034d</span><span class="o">:</span><span class="w"> </span><span class="nt">TRANSFER_TOKENS</span>
<span class="w"> </span><span class="nt">0x031b</span><span class="o">:</span><span class="w"> </span><span class="nt">CONS</span>
</code></pre></div>
<h2 id="setting-the-delegate"><a class="toclink" href="#setting-the-delegate">Setting the delegate</a></h2>
<p>Setting a delegate for a manager.tz contract is very similar; only the lambda needs to be changed to use the Michelson <code>SET_DELEGATE</code> instruction.</p>
<p>The lambda to send to the smart contract is</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"> </span><span class="kr">DROP</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="kt">operation</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">key_hash</span><span class="w"> </span><span class="err"><</span><span class="kt">delegate</span><span class="err">></span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">SOME</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">SET_DELEGATE</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">CONS</span><span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>The binary version of the operation is:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="o"><</span><span class="mi">32</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="nv">block</span><span class="w"> </span><span class="nv">hash</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">current</span><span class="w"> </span><span class="nv">head</span><span class="w"> </span><span class="nv">block</span><span class="w"> </span>[<span class="ss">(</span><span class="nv">see</span><span class="w"> </span><span class="nv">below</span><span class="ss">)</span>]<span class="ss">(</span>#<span class="nv">getting</span><span class="o">-</span><span class="nv">information</span><span class="o">-</span><span class="nv">to</span><span class="o">-</span><span class="nv">craft</span><span class="o">-</span><span class="nv">the</span><span class="o">-</span><span class="nv">operations</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x6c</span>:<span class="w"> </span><span class="nv">Transaction</span><span class="w"> </span><span class="nv">tag</span><span class="w"> </span><span class="ss">(</span><span class="mi">108</span><span class="ss">)</span>
<span class="w"> </span><span class="o"><</span><span class="mi">21</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="nv">key</span><span class="w"> </span><span class="nv">hash</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">source</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">transaction</span><span class="w"> </span><span class="ss">(</span><span class="nv">must</span><span class="w"> </span><span class="nv">be</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">manager</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">smart</span><span class="w"> </span><span class="nv">contract</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">xfd16</span>:<span class="w"> </span><span class="nv">fees</span><span class="w"> </span><span class="ss">(</span><span class="mi">2941</span><span class="w"> </span>μꜩ<span class="ss">)</span>
<span class="w"> </span><span class="o"><</span><span class="nv">counter</span><span class="o">></span>:<span class="w"> </span><span class="nv">counter</span><span class="w"> </span><span class="nv">associated</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">manager</span><span class="w"> </span><span class="nv">tz</span><span class="w"> </span><span class="nv">account</span><span class="w"> </span>[<span class="ss">(</span><span class="nv">see</span><span class="w"> </span><span class="nv">below</span><span class="ss">)</span>]<span class="ss">(</span>#<span class="nv">getting</span><span class="o">-</span><span class="nv">information</span><span class="o">-</span><span class="nv">to</span><span class="o">-</span><span class="nv">craft</span><span class="o">-</span><span class="nv">the</span><span class="o">-</span><span class="nv">operations</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">xabcd01</span>:<span class="w"> </span><span class="nv">gas_limit</span><span class="w"> </span><span class="ss">(</span><span class="mi">26283</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x00</span>:<span class="w"> </span><span class="nv">storage_limit</span><span class="w"> </span><span class="ss">(</span><span class="mi">0</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x00</span>:<span class="w"> </span><span class="nv">amount</span><span class="w"> </span><span class="ss">(</span><span class="mi">0</span>ꜩ<span class="ss">)</span>
<span class="w"> </span><span class="o"><</span><span class="mi">22</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">address</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">smart</span><span class="w"> </span><span class="nv">contract</span><span class="w"> </span><span class="ss">(</span><span class="nv">must</span><span class="w"> </span><span class="nv">start</span><span class="w"> </span><span class="nv">with</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">byte</span><span class="w"> </span><span class="mi">0</span><span class="nv">x01</span><span class="w"> </span><span class="nv">and</span><span class="w"> </span><span class="k">end</span><span class="w"> </span><span class="nv">with</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">byte</span><span class="w"> </span><span class="mi">0</span><span class="nv">x00</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">xff</span><span class="w"> </span><span class="ss">(</span><span class="nv">or</span><span class="w"> </span><span class="nv">any</span><span class="w"> </span><span class="nv">other</span><span class="w"> </span><span class="nv">non</span><span class="o">-</span><span class="nv">null</span><span class="w"> </span><span class="nv">byte</span><span class="ss">)</span>:<span class="w"> </span><span class="nv">presence</span><span class="w"> </span><span class="nv">flag</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">parameters</span><span class="w"> </span><span class="ss">(</span><span class="nv">entrypoint</span><span class="w"> </span><span class="nv">and</span><span class="w"> </span><span class="nv">argument</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x02</span>:<span class="w"> </span><span class="nv">tag</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="s2">"%do"</span><span class="w"> </span><span class="nv">entrypoint</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x0000002f</span>:<span class="w"> </span><span class="nv">length</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">argument</span><span class="w"> </span><span class="ss">(</span><span class="mi">47</span><span class="w"> </span><span class="nv">bytes</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x02</span>:<span class="w"> </span><span class="nv">Michelson</span><span class="w"> </span><span class="nv">sequence</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x0000002a</span>:<span class="w"> </span><span class="nv">length</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">sequence</span><span class="w"> </span><span class="ss">(</span><span class="mi">42</span><span class="w"> </span><span class="nv">bytes</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x0320</span>:<span class="w"> </span><span class="nv">DROP</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x053d</span>:<span class="w"> </span><span class="nv">NIL</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x036d</span>:<span class="w"> </span><span class="nv">operation</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x0743</span>:<span class="w"> </span><span class="nv">PUSH</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x035d</span>:<span class="w"> </span><span class="nv">key_hash</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x0a</span>:<span class="w"> </span><span class="nv">Byte</span><span class="w"> </span><span class="nv">sequence</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x00000015</span>:<span class="w"> </span><span class="nv">Length</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">sequence</span><span class="w"> </span><span class="ss">(</span><span class="mi">21</span><span class="w"> </span><span class="nv">bytes</span><span class="ss">)</span>
<span class="w"> </span><span class="o"><</span><span class="mi">21</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="o"><</span><span class="nv">destination</span><span class="o">></span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x0346</span>:<span class="w"> </span><span class="nv">SOME</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x034e</span>:<span class="w"> </span><span class="nv">SET_DELEGATE</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x031b</span>:<span class="w"> </span><span class="nv">CONS</span>
</code></pre></div>
<h2 id="removing-the-delegate"><a class="toclink" href="#removing-the-delegate">Removing the delegate</a></h2>
<p>Similarly, the delegate of the smart contract can be removed using the following lambda:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"> </span><span class="kr">DROP</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="kt">operation</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">NONE</span><span class="w"> </span><span class="kt">key_hash</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">SET_DELEGATE</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">CONS</span><span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>The binary version of the operation is:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="o"><</span><span class="mi">32</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="nv">block</span><span class="w"> </span><span class="nv">hash</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">current</span><span class="w"> </span><span class="nv">head</span><span class="w"> </span><span class="nv">block</span><span class="w"> </span>[<span class="ss">(</span><span class="nv">see</span><span class="w"> </span><span class="nv">below</span><span class="ss">)</span>]<span class="ss">(</span>#<span class="nv">getting</span><span class="o">-</span><span class="nv">information</span><span class="o">-</span><span class="nv">to</span><span class="o">-</span><span class="nv">craft</span><span class="o">-</span><span class="nv">the</span><span class="o">-</span><span class="nv">operations</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x6c</span>:<span class="w"> </span><span class="nv">Transaction</span><span class="w"> </span><span class="nv">tag</span><span class="w"> </span><span class="ss">(</span><span class="mi">108</span><span class="ss">)</span>
<span class="w"> </span><span class="o"><</span><span class="mi">21</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="nv">key</span><span class="w"> </span><span class="nv">hash</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">source</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">transaction</span><span class="w"> </span><span class="ss">(</span><span class="nv">must</span><span class="w"> </span><span class="nv">be</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">manager</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">smart</span><span class="w"> </span><span class="nv">contract</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">xfd16</span>:<span class="w"> </span><span class="nv">fees</span><span class="w"> </span><span class="ss">(</span><span class="mi">2941</span><span class="w"> </span>μꜩ<span class="ss">)</span>
<span class="w"> </span><span class="o"><</span><span class="nv">counter</span><span class="o">></span>:<span class="w"> </span><span class="nv">counter</span><span class="w"> </span><span class="nv">associated</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">manager</span><span class="w"> </span><span class="nv">tz</span><span class="w"> </span><span class="nv">account</span><span class="w"> </span>[<span class="ss">(</span><span class="nv">see</span><span class="w"> </span><span class="nv">below</span><span class="ss">)</span>]<span class="ss">(</span>#<span class="nv">getting</span><span class="o">-</span><span class="nv">information</span><span class="o">-</span><span class="nv">to</span><span class="o">-</span><span class="nv">craft</span><span class="o">-</span><span class="nv">the</span><span class="o">-</span><span class="nv">operations</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">xabcd01</span>:<span class="w"> </span><span class="nv">gas_limit</span><span class="w"> </span><span class="ss">(</span><span class="mi">26283</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x00</span>:<span class="w"> </span><span class="nv">storage_limit</span><span class="w"> </span><span class="ss">(</span><span class="mi">0</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x00</span>:<span class="w"> </span><span class="nv">amount</span><span class="w"> </span><span class="ss">(</span><span class="mi">0</span>ꜩ<span class="ss">)</span>
<span class="w"> </span><span class="o"><</span><span class="mi">22</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">address</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">smart</span><span class="w"> </span><span class="nv">contract</span><span class="w"> </span><span class="ss">(</span><span class="nv">must</span><span class="w"> </span><span class="nv">start</span><span class="w"> </span><span class="nv">with</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">byte</span><span class="w"> </span><span class="mi">0</span><span class="nv">x01</span><span class="w"> </span><span class="nv">and</span><span class="w"> </span><span class="k">end</span><span class="w"> </span><span class="nv">with</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">byte</span><span class="w"> </span><span class="mi">0</span><span class="nv">x00</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">xff</span><span class="w"> </span><span class="ss">(</span><span class="nv">or</span><span class="w"> </span><span class="nv">any</span><span class="w"> </span><span class="nv">other</span><span class="w"> </span><span class="nv">non</span><span class="o">-</span><span class="nv">null</span><span class="w"> </span><span class="nv">byte</span><span class="ss">)</span>:<span class="w"> </span><span class="nv">presence</span><span class="w"> </span><span class="nv">flag</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">parameters</span><span class="w"> </span><span class="ss">(</span><span class="nv">entrypoint</span><span class="w"> </span><span class="nv">and</span><span class="w"> </span><span class="nv">argument</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x02</span>:<span class="w"> </span><span class="nv">tag</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="s2">"%do"</span><span class="w"> </span><span class="nv">entrypoint</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x00000013</span>:<span class="w"> </span><span class="nv">length</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">argument</span><span class="w"> </span><span class="ss">(</span><span class="mi">19</span><span class="w"> </span><span class="nv">bytes</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x02</span>:<span class="w"> </span><span class="nv">Michelson</span><span class="w"> </span><span class="nv">sequence</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x0000000e</span>:<span class="w"> </span><span class="nv">length</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">sequence</span><span class="w"> </span><span class="ss">(</span><span class="mi">14</span><span class="w"> </span><span class="nv">bytes</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x0320</span>:<span class="w"> </span><span class="nv">DROP</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x053d</span>:<span class="w"> </span><span class="nv">NIL</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x036d</span>:<span class="w"> </span><span class="nv">operation</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x053e</span>:<span class="w"> </span><span class="nv">NONE</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x035d</span>:<span class="w"> </span><span class="nv">key_hash</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x034e</span>:<span class="w"> </span><span class="nv">SET_DELEGATE</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x031b</span>:<span class="w"> </span><span class="nv">CONS</span>
</code></pre></div>
<h2 id="origination-of-the-manager-script"><a class="toclink" href="#origination-of-the-manager-script">Origination of the manager script</a></h2>
<p>To originate a smart contract running the manager.tz script, we need
to send an “origination” operation with the following data:</p>
<ul>
<li>
<p><span class="dquo">“</span>source”, “counter”, “balance”, and “delegate” with the same values
as a <span class="caps">KT1</span> account origination in Athens</p>
</li>
<li>
<p><span class="dquo">“</span>gas_limit”: at least 15555 gas units</p>
</li>
<li>
<p><span class="dquo">“</span>storage_limit”: at least 489 bytes</p>
</li>
<li>
<p><span class="dquo">“</span>storage”: the (Michelson representation of the) key_hash of the manager</p>
</li>
<li>
<p><span class="dquo">“</span>script”: the manager script</p>
</li>
<li>
<p>JSon version:</p>
</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="p">{</span> <span class="s">"code"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"parameter"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"or"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"lambda"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"unit"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"list"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"operation"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">}</span> <span class="p">],</span>
<span class="s">"annots"</span><span class="p">:</span> <span class="p">[</span> <span class="s">"%do"</span> <span class="p">]</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"unit"</span><span class="p">,</span> <span class="s">"annots"</span><span class="p">:</span> <span class="p">[</span> <span class="s">"%default"</span> <span class="p">]</span> <span class="p">}</span> <span class="p">]</span> <span class="p">}</span> <span class="p">]</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"storage"</span><span class="p">,</span> <span class="s">"args"</span><span class="p">:</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"key_hash"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"code"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">[</span> <span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"DUP"</span> <span class="p">},</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"CAR"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"DIP"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span> <span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"CDR"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">]</span> <span class="p">}</span> <span class="p">]</span> <span class="p">],</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"IF_LEFT"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"PUSH"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"mutez"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"int"</span><span class="p">:</span> <span class="s">"0"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"AMOUNT"</span> <span class="p">},</span>
<span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"COMPARE"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"EQ"</span> <span class="p">}</span> <span class="p">],</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"IF"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">[],</span>
<span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"UNIT"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"FAILWITH"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">]</span> <span class="p">]</span> <span class="p">}</span> <span class="p">],</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"DIP"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span> <span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"DUP"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">]</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"SWAP"</span> <span class="p">}</span> <span class="p">],</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"IMPLICIT_ACCOUNT"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"ADDRESS"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"SENDER"</span> <span class="p">},</span>
<span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"COMPARE"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"EQ"</span> <span class="p">}</span> <span class="p">],</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"IF"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">[],</span>
<span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"UNIT"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"FAILWITH"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">]</span> <span class="p">]</span> <span class="p">}</span> <span class="p">],</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"UNIT"</span> <span class="p">},</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"EXEC"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"PAIR"</span> <span class="p">}</span> <span class="p">],</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"DROP"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"NIL"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"operation"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"PAIR"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">]</span> <span class="p">}</span> <span class="p">]</span> <span class="p">]</span> <span class="p">}</span>
</code></pre></div>
<ul>
<li>Michelson version:</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="k">parameter</span>
<span class="w"> </span><span class="p">(</span><span class="kt">or</span>
<span class="w"> </span><span class="p">(</span><span class="kt">lambda</span><span class="w"> </span><span class="k k-Name">%do</span><span class="w"> </span><span class="kt">unit</span><span class="w"> </span><span class="p">(</span><span class="kt">list</span><span class="w"> </span><span class="kt">operation</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">unit</span><span class="w"> </span><span class="k k-Name">%default</span><span class="p">));</span>
<span class="k">storage</span><span class="w"> </span><span class="kt">key_hash</span><span class="p">;</span>
<span class="k">code</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">UNPAIR</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">IF_LEFT</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="c1"># 'do' entrypoint</span>
<span class="w"> </span><span class="c1"># Assert no token was sent:</span>
<span class="w"> </span><span class="c1"># to send tokens, the default entry point should be used</span>
<span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">mutez</span><span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">AMOUNT</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">ASSERT_CMPEQ</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="c1"># Assert that the sender is the manager</span>
<span class="w"> </span><span class="kr">DUUP</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">IMPLICIT_ACCOUNT</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">ADDRESS</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">SENDER</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">ASSERT_CMPEQ</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="c1"># Execute the lambda argument</span>
<span class="w"> </span><span class="kr">UNIT</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">EXEC</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">PAIR</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="c1"># 'default' entrypoint</span>
<span class="w"> </span><span class="kr">DROP</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="kt">operation</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">PAIR</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">};</span>
</code></pre></div>
<ul>
<li>Binary version (including the initial 4-byte size):</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="mf">0</span><span class="n">x000000c602000000c105000764085e036c055f036d0000000325646f046c000000082564656661756c740501035d050202000000950200000012020000000d03210316051f02000000020317072e020000006a0743036a00000313020000001e020000000403190325072c020000000002000000090200000004034f0327020000000b051f02000000020321034c031e03540348020000001e020000000403190325072c020000000002000000090200000004034f0327034f0326034202000000080320053d036d0342</span>
</code></pre></div>
<p>The binary format for this operation is as follows:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="o"><</span><span class="mi">32</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="nv">block</span><span class="w"> </span><span class="nv">hash</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">current</span><span class="w"> </span><span class="nv">head</span><span class="w"> </span><span class="nv">block</span><span class="w"> </span>[<span class="ss">(</span><span class="nv">see</span><span class="w"> </span><span class="nv">below</span><span class="ss">)</span>]<span class="ss">(</span>#<span class="nv">getting</span><span class="o">-</span><span class="nv">information</span><span class="o">-</span><span class="nv">to</span><span class="o">-</span><span class="nv">craft</span><span class="o">-</span><span class="nv">the</span><span class="o">-</span><span class="nv">operations</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x6d</span>:<span class="w"> </span><span class="nv">Origination</span><span class="w"> </span><span class="nv">tag</span><span class="w"> </span><span class="ss">(</span><span class="mi">108</span><span class="ss">)</span>
<span class="w"> </span><span class="o"><</span><span class="mi">21</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="nv">key</span><span class="w"> </span><span class="nv">hash</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">source</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">transaction</span><span class="w"> </span><span class="ss">(</span><span class="nv">must</span><span class="w"> </span><span class="nv">be</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">hash</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">key</span><span class="w"> </span><span class="nv">that</span><span class="w"> </span><span class="nv">will</span><span class="w"> </span><span class="nv">sign</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">operation</span>,<span class="w"> </span><span class="nv">not</span><span class="w"> </span><span class="nv">necesseraly</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">manager</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x8510</span>:<span class="w"> </span><span class="nv">fees</span><span class="w"> </span><span class="ss">(</span><span class="mi">2053</span><span class="w"> </span>μꜩ<span class="ss">)</span>
<span class="w"> </span><span class="o"><</span><span class="nv">counter</span><span class="o">></span>:<span class="w"> </span><span class="nv">counter</span><span class="w"> </span><span class="nv">associated</span><span class="w"> </span><span class="nv">to</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">manager</span><span class="w"> </span><span class="nv">tz</span><span class="w"> </span><span class="nv">account</span><span class="w"> </span>[<span class="ss">(</span><span class="nv">see</span><span class="w"> </span><span class="nv">below</span><span class="ss">)</span>]<span class="ss">(</span>#<span class="nv">getting</span><span class="o">-</span><span class="nv">information</span><span class="o">-</span><span class="nv">to</span><span class="o">-</span><span class="nv">craft</span><span class="o">-</span><span class="nv">the</span><span class="o">-</span><span class="nv">operations</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">xbe7a</span>:<span class="w"> </span><span class="nv">gas_limit</span><span class="w"> </span><span class="ss">(</span><span class="mi">15678</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">xfd03</span>:<span class="w"> </span><span class="nv">storage_limit</span><span class="w"> </span><span class="ss">(</span><span class="mi">509</span><span class="w"> </span><span class="nv">bytes</span><span class="ss">)</span>
<span class="w"> </span><span class="o"><</span><span class="nv">init</span><span class="w"> </span><span class="nv">balance</span><span class="o">></span>:<span class="w"> </span><span class="nv">initial</span><span class="w"> </span><span class="nv">balance</span><span class="w"> </span><span class="nv">in</span><span class="w"> </span>μꜩ
<span class="w"> </span><span class="mi">0</span><span class="nv">xff</span><span class="w"> </span><span class="ss">(</span><span class="nv">or</span><span class="w"> </span><span class="nv">any</span><span class="w"> </span><span class="nv">other</span><span class="w"> </span><span class="nv">non</span><span class="o">-</span><span class="nv">null</span><span class="w"> </span><span class="nv">byte</span><span class="ss">)</span>:<span class="w"> </span><span class="nv">presence</span><span class="w"> </span><span class="nv">flag</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">delegate</span>
<span class="w"> </span><span class="o"><</span><span class="mi">21</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="nv">key</span><span class="w"> </span><span class="nv">hash</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">initial</span><span class="w"> </span><span class="nv">delegate</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x000000c602000000c105000764085e036c055f036d0000000325646f046c000000082564656661756c740501035d050202000000950200000012020000000d03210316051f02000000020317072e020000006a0743036a00000313020000001e020000000403190325072c020000000002000000090200000004034f0327020000000b051f02000000020321034c031e03540348020000001e020000000403190325072c020000000002000000090200000004034f0327034f0326034202000000080320053d036d0342</span>:<span class="w"> </span><span class="nv">manager</span><span class="w"> </span><span class="nv">script</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x0000001a</span>:<span class="w"> </span><span class="nv">storage</span><span class="w"> </span><span class="nv">length</span><span class="w"> </span><span class="ss">(</span><span class="mi">26</span><span class="w"> </span><span class="nv">bytes</span><span class="ss">)</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x0a</span>:<span class="w"> </span><span class="nv">storage</span><span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">byte</span><span class="w"> </span><span class="nv">sequence</span>
<span class="w"> </span><span class="mi">0</span><span class="nv">x00000015</span>:<span class="w"> </span><span class="nv">length</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">sequence</span><span class="w"> </span><span class="ss">(</span><span class="mi">21</span><span class="w"> </span><span class="nv">bytes</span><span class="ss">)</span>
<span class="w"> </span><span class="o"><</span><span class="mi">21</span><span class="w"> </span><span class="nv">bytes</span><span class="o">></span>:<span class="w"> </span><span class="nv">storage</span>,<span class="w"> </span><span class="nv">this</span><span class="w"> </span><span class="nv">is</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">key</span><span class="w"> </span><span class="nv">hash</span><span class="w"> </span><span class="nv">of</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">smart</span><span class="w"> </span><span class="nv">contract</span><span class="err">'s manager</span>
</code></pre></div>
<p>We can then sign this sequence of bytes with the sender’s private key and send the json version of the operation through an <span class="caps">RPC</span></p>
<div class="highlight"><pre><span></span><code><span class="o">/</span><span class="n">chains</span><span class="o">/</span><span class="n">main</span><span class="o">/</span><span class="n">blocks</span><span class="o">/</span><span class="n">head</span><span class="o">/</span><span class="n">helpers</span><span class="o">/</span><span class="n">preapply</span><span class="o">/</span><span class="n">operations</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"protocol"</span><span class="p">:</span> <span class="s">"PsBABY5HQTSkA4297zNHfsZNKtxULfL18y95qb3m53QJiXGmrbU"</span><span class="p">,</span>
<span class="s">"branch"</span><span class="p">:</span> <span class="s">"<block hash of the head block in Base58 [(see below)](#getting-information-to-craft-the-operations)>"</span><span class="p">,</span>
<span class="s">"contents"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"kind"</span><span class="p">:</span> <span class="s">"origination"</span><span class="p">,</span>
<span class="s">"source"</span><span class="p">:</span> <span class="s">"<sender>"</span><span class="p">,</span> <span class="s">"fee"</span><span class="p">:</span> <span class="s">"2053"</span><span class="p">,</span>
<span class="s">"counter"</span><span class="p">:</span> <span class="s">"<counter [(see below)](#getting-information-to-craft-the-operations)>"</span><span class="p">,</span> <span class="s">"gas_limit"</span><span class="p">:</span> <span class="s">"15678"</span><span class="p">,</span>
<span class="s">"storage_limit"</span><span class="p">:</span> <span class="s">"509"</span><span class="p">,</span> <span class="s">"balance"</span><span class="p">:</span> <span class="s">"<balance>"</span><span class="p">,</span>
<span class="s">"script"</span><span class="p">:</span>
<span class="p">{</span> <span class="s">"code"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"parameter"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"or"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"lambda"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"unit"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"list"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"operation"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">}</span> <span class="p">],</span>
<span class="s">"annots"</span><span class="p">:</span> <span class="p">[</span> <span class="s">"%do"</span> <span class="p">]</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"unit"</span><span class="p">,</span> <span class="s">"annots"</span><span class="p">:</span> <span class="p">[</span> <span class="s">"%default"</span> <span class="p">]</span> <span class="p">}</span> <span class="p">]</span> <span class="p">}</span> <span class="p">]</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"storage"</span><span class="p">,</span> <span class="s">"args"</span><span class="p">:</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"key_hash"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"code"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">[</span> <span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"DUP"</span> <span class="p">},</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"CAR"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"DIP"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span> <span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"CDR"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">]</span> <span class="p">}</span> <span class="p">]</span> <span class="p">],</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"IF_LEFT"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"PUSH"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"mutez"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"int"</span><span class="p">:</span> <span class="s">"0"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"AMOUNT"</span> <span class="p">},</span>
<span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"COMPARE"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"EQ"</span> <span class="p">}</span> <span class="p">],</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"IF"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">[],</span>
<span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"UNIT"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"FAILWITH"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">]</span> <span class="p">]</span> <span class="p">}</span> <span class="p">],</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"DIP"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span> <span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"DUP"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">]</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"SWAP"</span> <span class="p">}</span> <span class="p">],</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"IMPLICIT_ACCOUNT"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"ADDRESS"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"SENDER"</span> <span class="p">},</span>
<span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"COMPARE"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"EQ"</span> <span class="p">}</span> <span class="p">],</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"IF"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span>
<span class="p">[</span> <span class="p">[],</span>
<span class="p">[</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"UNIT"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"FAILWITH"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">]</span> <span class="p">]</span> <span class="p">}</span> <span class="p">],</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"UNIT"</span> <span class="p">},</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"EXEC"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"PAIR"</span> <span class="p">}</span> <span class="p">],</span>
<span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"DROP"</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"NIL"</span><span class="p">,</span>
<span class="s">"args"</span><span class="p">:</span> <span class="p">[</span> <span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"operation"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">},</span>
<span class="p">{</span> <span class="s">"prim"</span><span class="p">:</span> <span class="s">"PAIR"</span> <span class="p">}</span> <span class="p">]</span> <span class="p">]</span> <span class="p">}</span> <span class="p">]</span> <span class="p">]</span> <span class="p">}</span> <span class="p">],</span>
<span class="s">"storage"</span><span class="p">:</span>
<span class="p">{</span> <span class="s">"bytes"</span><span class="p">:</span> <span class="s">"<manager's key hash in hexadecimal without the leading 0x>"</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> <span class="p">],</span>
<span class="s">"signature"</span><span class="p">:</span> <span class="s">"<signature>"</span> <span class="p">}</span> <span class="p">]</span>
</code></pre></div>
<h2 id="getting-informations-to-craft-the-operations"><a class="toclink" href="#getting-informations-to-craft-the-operations">Getting informations to craft the operations</a></h2>
<p>To craft the operations described in the previous sections, you will
need to query a synchronised node as follow:</p>
<ul>
<li>
<p>The block hash of the current head block is obtained
from the <span class="caps">RPC</span> <code>GET /chains/main/blocks/head/hash</code></p>
</li>
<li>
<p>the counter <code><counter></code> associated to a <code><tz...></code> account is
obtained from the <span class="caps">RPC</span> <code>GET
/chains/main/blocks/head/context/contracts/<tz...>/counter</code>.</p>
</li>
</ul>
<h2 id="gas-cost-recap"><a class="toclink" href="#gas-cost-recap">Gas cost recap</a></h2>
<p>Due to the changes brought by Babylon, gas costs will change and will
probably change again in the future. Ideally, the gas cost of a transaction
should be queried from a trusted node or indexer. For convenience, we list
recommended gas costs for the most important cases below.</p>
<ul>
<li>implicit (tz) to implicit: 10307</li>
<li>implicit to manager.tz: 15385</li>
<li>manager.tz to implicit: 26283</li>
<li>manager.tz to manager.tz: 44725</li>
</ul>
<h2 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h2>
<p>The Babylon update simplifies the organisation of Tezos accounts by
removing the “delegatable” and “spendable” flags together with the
“manager” address of <span class="caps">KT1</span> accounts and smart contracts. The interaction
of these flags with smart contract codes were sometimes difficult to
grasp. All modifications to <span class="caps">KT1</span> states (balance, delegate, or storage)
now have to go through the smart contract code. These breaking changes
impact the development of wallet application but come with the new
feature of delegating tz account which was developed to simplify the
delegation workflow since originating a <span class="caps">KT1</span> account is no more
necessary to delegate.</p>Babylon: Proposal Injected!2019-07-26T14:00:00+02:002019-07-26T14:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2019-07-26:/babylon-proposal-injected.html<p>Summary:</p>
<p><b><span class="caps">EDIT</span> (August 2, 2019)</b>: The updated Babylon proposal has been injected
by Cryptium Labs. Its hash is <code>PsBABY5HQTSkA4297zNHfsZNKtxULfL18y95qb3m53QJiXGmrbU</code>.
The instructions contained in this post have been updated accordingly.</p>
<hr>
<p>Cryptium Labs just injected the hash of our new joint proposal: <code>Babylon</code>.
This triggers the beginning of the third on-chain vote …</p><p>Summary:</p>
<p><b><span class="caps">EDIT</span> (August 2, 2019)</b>: The updated Babylon proposal has been injected
by Cryptium Labs. Its hash is <code>PsBABY5HQTSkA4297zNHfsZNKtxULfL18y95qb3m53QJiXGmrbU</code>.
The instructions contained in this post have been updated accordingly.</p>
<hr>
<p>Cryptium Labs just injected the hash of our new joint proposal: <code>Babylon</code>.
This triggers the beginning of the third on-chain vote to amend Tezos.
This process could end in the successful migration from current
protocol <code>Athens</code> in about three months, if the participants decide so.</p>
<p>This update is joint work with Cryptium Labs, with contributions
from the Marigold team, and it is much more significant than <code>Athens</code>
in terms of features.</p>
<p>Its main changes to Tezos are:</p>
<ul>
<li>a new variant of our consensus algorithm, Emmy+, that is both more
robust and simpler to analyse,</li>
<li>many new Michelson features which have been eagerly awaited by Tezos smart
contract developers and designers of higher level languages,</li>
<li>an account rehaul that establishes a clear distinction between
accounts (tz1, tz2 and tz3) and smart contracts (<span class="caps">KT1</span>),</li>
<li>refinements to the quorum formula and a new 5% proposal quorum.</li>
</ul>
<p>This time, we made only one proposal, as we believe that all features
in it are necessary improvements for the future of Tezos.</p>
<p>Like last time, this proposal includes a symbolic invoice of ꜩ500.
For a twist, we credited an instance of our verified multisig
shared among the contributors to this proposal to pay for drinks.</p>
<p><img alt="A Babylonian signing roll" src="images/babylon.svg" style="display:block;float:right;width:25%;"></p>
<p>This post includes instructions to download and observe the code, to
recompute the hash by yourself, and to examine the differences with
the current protocol.
It then gives the method we recommend to bakers if they wish to upvote
the proposal.</p>
<p>Beyond these on-chain administrative details, the post summarises the
main changes, giving simple high level explanations and pointers to
more detailed articles on our blog or Cryptium Labs’.</p>
<p>We also provide a <a href="http://tezos.gitlab.io/protocols/005_PsBABY5H.html">reference documentation
page</a> for the
proposal which contains information on breaking changes for node
administrators and application developers (to be completed soon).
This page also contains a curated subset of the changelog that
everybody is strongly encouraged to read carefully.</p>
<p>Finally, bear with us until the end for a geeky bonus.</p>
<h1 id="getting-the-source-code-of-the-proposals"><a class="toclink" href="#getting-the-source-code-of-the-proposals">Getting the source code of the proposals</a></h1>
<p>The source code of this proposal is available in
<a href="files/babylon_005_PsBABY5H.tar">this tar archive</a>.</p>
<p>To compute the hash, you can use the <code>tezos-protocol-compiler</code> as
follows. The result should be <code>PsBABY5HQTSkA4297zNHfsZNKtxULfL18y95qb3m53QJiXGmrbU</code>.</p>
<div class="highlight"><pre><span></span><code>mkdir proposal && cd proposal
curl https://research-development.nomadic-labs.com/files/babylon_005_PsBABY5H.tar | tar -x
tezos-protocol-compiler -hash-only babylon_005_PsBABY5H/lib_protocol
</code></pre></div>
<p>If you want to directly investigate how Babylon differs from the current
protocol, you can use <code>diff</code> against folder <code>proto_004_Pt24m4xi</code> of the
mainnet source code. All the changes should be the ones detailed in the <em>changes</em>
section, except for the removal of the single use migration code from
the previous upgrade (from <code>003_PsddFKi3</code> to <code>proto_004_Pt24m4xi</code>).</p>
<h1 id="how-to-upvote-the-proposal"><a class="toclink" href="#how-to-upvote-the-proposal">How to upvote the proposal</a></h1>
<p>The easiest way to create and sign a voting operation today is to use
the command line Tezos client, as follows, and as detailed in the
<a href="http://tezos.gitlab.io/whitedoc/voting.html#client-commands">online
documentation</a>.</p>
<p>If you choose to upvote the proposal:</p>
<p><code>tezos-client submit proposals for <delegate> PsBABY5HQTSkA4297zNHfsZNKtxULfL18y95qb3m53QJiXGmrbU</code></p>
<h1 id="detailed-list-of-changes"><a class="toclink" href="#detailed-list-of-changes">Detailed list of changes</a></h1>
<h2 id="emmy"><a class="toclink" href="#emmy">Emmy+</a></h2>
<p>Protocol 004 implements a consensus algorithm nick-named
<code>Emmy</code>.
Protocol 005 introduces several improvements to this algorithm,
regrouped under the name <code>Emmy+</code>.
In particular:</p>
<ul>
<li>the fewer endorsements a block carries the longer it takes before it
can be considered valid,</li>
<li>the fitness of a block is simply its height in the chain,</li>
<li>changes to the rewards for block and endorsement of priority greater
than 0.</li>
</ul>
<p>Detailed informations can be found in the blog
<a href="https://research-development.nomadic-labs.com/emmy-an-improved-consensus-algorithm.html">announcement</a>
and
<a href="https://research-development.nomadic-labs.com/analysis-of-emmy.html">analysis</a>.</p>
<p>Merge requests
<a href="https://gitlab.com/nomadic-labs/tezos/-/merge_requests/58">1</a>
and
<a href="https://gitlab.com/nomadic-labs/tezos/-/merge_requests/72">2</a>.</p>
<h2 id="michelson"><a class="toclink" href="#michelson">Michelson</a></h2>
<p>Protocol 005 contains several improvements to the Michelson smart
contract language.
More details are provided later in the changelog and in the
<a href="https://research-development.nomadic-labs.com/michelson-updates-in-005.html">Michelson update blog
post</a>.</p>
<p>A summary of the main changes:</p>
<ul>
<li>smart contracts now support entrypoints
<a href="https://gitlab.com/nomadic-labs/tezos/-/merge_requests/59">(<span class="caps">MR</span>)</a>,</li>
<li>contracts can now create, store and transmit as many <code>big_map</code>s as
they want
<a href="https://gitlab.com/nomadic-labs/tezos/-/merge_requests/76">(<span class="caps">MR</span>)</a>,</li>
<li>comparable types are now closed under products (i.e. the pair constructor),</li>
<li>a new instruction, <code>CHAIN_ID</code>, allows contracts to differentiate
between the test chain and the main network
<a href="https://gitlab.com/nomadic-labs/tezos/-/merge_requests/65">(<span class="caps">MR</span>)</a>,</li>
<li>a gas cost overhaul has been integrated, and <code>STEPS_TO_QUOTA</code> has
been disabled until a more robust semantics is found
<a href="https://gitlab.com/nomadic-labs/tezos/-/merge_requests/73">(<span class="caps">MR</span>)</a>.</li>
</ul>
<h3 id="new-instructions-to-facilitate-compilation-to-michelson-marigold"><a class="toclink" href="#new-instructions-to-facilitate-compilation-to-michelson-marigold">New instructions to facilitate compilation to Michelson - Marigold</a></h3>
<p>Some new instructions have been added in order to make Michelson a
better compilation target for high level languages.</p>
<p>The new instructions <code>DIG n</code>, <code>DUG n</code>, <code>DIP n { code }</code>, <code>DROP n</code> allow to
simplify commonly used patterns
<a href="https://gitlab.com/nomadic-labs/tezos/-/merge_requests/81">(<span class="caps">MR</span>)</a>.</p>
<p>The new instruction <code>APPLY</code> allows to perform the partial application of
an argument to a lambda
<a href="https://gitlab.com/nomadic-labs/tezos/-/merge_requests/46">(<span class="caps">MR</span>)</a>.</p>
<h3 id="changes-for-compatibility-with-the-accounts-rehaul"><a class="toclink" href="#changes-for-compatibility-with-the-accounts-rehaul">Changes for compatibility with the accounts rehaul</a></h3>
<p>A few changes are needed as a consequence of the accounts simplification.</p>
<ul>
<li>the instruction <code>CREATE_ACCOUNT</code> disappears,</li>
<li>the type of <code>CREATE_CONTRACT</code> changes.</li>
</ul>
<h3 id="proposal-quorum-cryptium-labs"><a class="toclink" href="#proposal-quorum-cryptium-labs">Proposal quorum - Cryptium Labs</a></h3>
<p>During the first period of a voting procedure, proposals can be
submitted and the one with the largest number of upvotes is promoted
to the testing vote period.
In 004 the promoted proposal can have any number of upvotes, even 1,
as long as the others proposals, if any, have less.
Protocol 005 introduces a quorum of 5% (constant
<code>min_proposal_quorum</code>) for proposals to be promoted to the testing
vote period, if there is less than 5% of the stake supporting them,
the protocol goes back to a proposal period.</p>
<p><a href="https://medium.com/cryptium/meanwhile-at-cryptium-labs-1-part-v-3e0ddfd98177">Cryptium’s blog post</a>.</p>
<p><a href="https://gitlab.com/nomadic-labs/tezos/-/merge_requests/71">Merge request</a>.</p>
<h3 id="quorum-caps-cryptium-labs"><a class="toclink" href="#quorum-caps-cryptium-labs">Quorum caps - Cryptium Labs</a></h3>
<p>During the test phases the participation needs to reach a quorum for a
vote to be successful.
The quorum adapts over time based on the participation of past votes.
In 004 the quorum can reach very high values which would make passing
new proposals very difficult even if there is large acceptance.
On the other hand the quorum could reach very low levels if there is
little participation.
Protocol 005 introduces caps to limit the maximum and minimum values
that the quorum can reach.
The values proposed for minimum quorum cap is set to 20% and the
maximum to 70%, these values can be changed in future updates.
Additionally the formula to update the quorum uses an exponential
moving average of the participation.</p>
<p><a href="https://medium.com/cryptium/meanwhile-at-cryptium-labs-1-part-ii-607227fc6d65">Cryptium’s blog post</a>.</p>
<p><a href="https://gitlab.com/nomadic-labs/tezos/-/merge_requests/52">Merge Request</a>.</p>
<h3 id="make-implicit-accounts-delegatable-cryptium-labs"><a class="toclink" href="#make-implicit-accounts-delegatable-cryptium-labs">Make implicit accounts delegatable - Cryptium Labs</a></h3>
<p>In protocol 004 only <span class="caps">KT1</span> addresses, representing an account for
delegation or a smart contract, can be delegated and only tz accounts
can register as delegate.
In protocol 005, tz accounts which are not registered as
delegate can be delegated towards a tz account registered as delegate.
This change does not affect existing delegations of <span class="caps">KT</span> accounts.</p>
<p>One restriction remains that may be lifted in the future: once a tz
account is registered as delegate it cannot be un-registered.
This in turn means that a registered delegate that wants to stop being
one, cannot delegate to somebody else.
The only solution for now is to move the funds to a newly created tz
account and delegate from there.</p>
<p>Cryptium’s blog posts
<a href="https://medium.com/cryptium/meanwhile-at-cryptium-labs-1-part-iii-1c824b760da3">1</a>
and
<a href="https://medium.com/cryptium/meanwhile-at-cryptium-labs-1-part-vi-540170f46c51">2</a>.</p>
<p><a href="https://gitlab.com/nomadic-labs/tezos/-/merge_requests/61">Merge request</a>.</p>
<h3 id="replace-kt1-accounts-with-managertz-script-cryptium-labs"><a class="toclink" href="#replace-kt1-accounts-with-managertz-script-cryptium-labs">Replace <span class="caps">KT1</span> accounts with <code>manager.tz</code> script - Cryptium Labs</a></h3>
<p>In 004 an address <span class="caps">KT1</span> can refer to a scriptless account used for
delegation or to a smart contract with code.
Given that in 005 it is possible to delegate from tz accounts,
scriptless <span class="caps">KT1</span> accounts are deprecated.
Existing <span class="caps">KT1</span> accounts are replaced with a smart contract
<code>manager.tz</code> which implements the same semantics.
The smart contract has been formally verified in Mi-Cho-Coq.</p>
<p>While the migrated accounts preserve all their features, this will
change the way wallets and other applications interact with them.
Detailed instructions for migrating such applications will be provided
in the coming days in the <a href="http://tezos.gitlab.io/protocols/005_PsBABY5H.html">documentation
page</a>.</p>
<p>Cryptium’s blog posts
<a href="https://medium.com/cryptium/meanwhile-at-cryptium-labs-1-part-iii-1c824b760da3">1</a>
and
<a href="https://medium.com/cryptium/meanwhile-at-cryptium-labs-1-part-vi-540170f46c51">2</a>.</p>
<p><a href="https://gitlab.com/nomadic-labs/mi-cho-coq/blob/master/src/contracts/manager.tz">manager.tz script</a>
and
<a href="https://gitlab.com/nomadic-labs/mi-cho-coq/blob/master/src/contracts_coq/manager.v">proof</a>.</p>
<p>Merge requests
<a href="https://gitlab.com/nomadic-labs/tezos/-/merge_requests/66">1</a>
and
<a href="https://gitlab.com/nomadic-labs/tezos/-/merge_requests/74">2</a>.</p>
<h1 id="bonus-section-how-did-we-get-this-pretty-protocol-hash"><a class="toclink" href="#bonus-section-how-did-we-get-this-pretty-protocol-hash">Bonus section: how did we get this pretty protocol hash ?</a></h1>
<p>The hash of the protocol is a unique identifier computed from its
source code. Each time we modify a character in the source code, the
hash changes. To obtain a specific hash, or a hash with a specific
form, the only solution is to generate a lot of source codes until the
hash matches.</p>
<p>This is what we did, by tweaking the Tezos compiler to include a small
(non significant) random part in <code>Babylon</code><span class="quo">‘</span>s source code, and detect
it the hash as rendered in Base58 starts with <code>baby</code>. If you look in
<code>main.ml</code>, you will see a comment line <code>(* Vanity nonce: 1260059503 *)</code>.</p>
<p>This cool trick is actually useful, as it will allow humans to quickly
identify the protocol from the hash when looking at <span class="caps">RPC</span> results.</p>
<p>You can do a similar thing with your Tezos addresses with the command
line client, by running command <code>gen vanity keys <new> matching
[<words>...] --prefix</code>. Of course this generates an encoded key on
your disk, which is not as secure as using a hardware wallet. Creating
a vanity key on such a device, is an exercise left to the reader.</p>Michelson updates in 0052019-07-25T18:00:00+02:002019-07-25T18:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2019-07-25:/michelson-updates-in-005.html<h1 id="changes-in-michelson"><a class="toclink" href="#changes-in-michelson">Changes in Michelson</a></h1>
<p>As hinted at in a
<a href="https://research-development.nomadic-labs.com/meanwhile-at-nomadic-labs-3.html">previous blog post</a>,
we’ve been working on improving different parts of the protocol, including our
favourite smart contract language: Michelson.</p>
<p>The changes made to Michelson in this proposal intend to simplify smart contract
development by making the code of complex contracts simpler and cleaner. In particular:</p>
<ul>
<li>smart contracts now support <strong>entrypoints</strong></li>
<li>contracts can now <strong>create, store and transmit as many big_maps</strong> as they want</li>
<li>comparable types are now closed under products (i.e. the pair constructor)</li>
<li>a new instruction, <code>CHAIN_ID</code>, allows contracts to differentiate between the test chain and the main network</li>
<li>a <strong>gas cost overhaul</strong> has been integrated, and <code>STEPS_TO_QUOTA</code> has been disabled until a more robust semantics is found.</li>
</ul>
<p>Some new instructions have been added in order to make Michelson a
better compilation target for high level languages:</p>
<ul>
<li>new stack instructions <code>DIG n</code>, <code>DUG n</code>, <code>DIP n { code }</code>, <code>DROP n</code></li>
<li>support for partial application and closures with the <code>APPLY</code> instruction</li>
</ul>
<p>Finally, a series of changes implement the new, cleaner, distinction between accounts and smart contracts as described in
<a href="https://medium.com/cryptium/meanwhile-at-cryptium-labs-1-part-vi-540170f46c51">Cryptium Labs’ blog</a>:</p>
<ul>
<li>the instruction <code>CREATE_ACCOUNT</code> disappears</li>
<li>the type of <code>CREATE_CONTRACT</code> changes, as a consequence of the amendments described in
<a href="https://medium.com/cryptium/meanwhile-at-cryptium-labs-1-part-vi-540170f46c51">Cryptium Labs’ blog</a></li>
<li>during the migration, the manager operations currently implemented at the protocol level will be automatically
converted into smart contract entrypoints.</li>
</ul>
<p>We’re going to go through these in more details, but before we
continue, you should know that some of these will introduce <span
style="color:red"><strong><span class="caps">BREAKING</span> <span class="caps">CHANGES</span></strong></span> for application
developers. The changes should <strong><span class="caps">NOT</span> break or radically change the
semantics of existing contracts</strong>. It may <strong><span class="caps">ONLY</span> change the interface
that they expose</strong>. As a consequence, shallow changes to the rest of
your stack may be needed.</p>
<p>In particular:</p>
<ul>
<li>some RPCs will expect different inputs and provide different outputs;</li>
<li>the binary representation of operations will change due to both
the account rehaul and entrypoints;</li>
<li>the introduction of multiple entrypoints can
give a special meaning to some existing annotations on arguments (such as <code>%default</code>),
changing the type of some pre-existing contracts;</li>
<li>some gas costs are updated.</li>
</ul>
<p><strong>If you maintain a tool relying on RPCs or binary formats, or are a
contract writer, please test your tools.</strong> We tried to
keep the changes as local and manageable as possible.</p>
<p>More details are provided throughout this article and we will make forthcoming
communications to aid this transition.</p>
<h1 id="multiple-entrypoints"><a class="toclink" href="#multiple-entrypoints">Multiple entrypoints</a></h1>
<p><em>Lightweight multiple entrypoints</em> are one of the most important new
features of this proposal, as they let you define extensible standard
interfaces for smart contracts. Concretely, entrypoints leverage the
annotation mechanism to call specific parts of the target smart contract
code in an easy way.</p>
<h2 id="usage"><a class="toclink" href="#usage">Usage</a></h2>
<p>Entrypoints correspond to annotations in the input type of a Michelson
contract, such as <code>parameter (or (int %set_age) (string %set_name))</code>.</p>
<p>Calling a contract at entrypoint <code>%set_age</code> with argument <code>12</code> is easy:</p>
<div class="highlight"><pre><span></span><code>tezos-client transfer <AMOUNT> from <CALLER> to <CONTRACT> --entrypoint set_age --arg 12
</code></pre></div>
<h2 id="overview-of-entrypoints"><a class="toclink" href="#overview-of-entrypoints">Overview of entrypoints</a></h2>
<p><em>Entrypoints</em> allow a contract to be seen as a collection of
independent functions sharing a storage, and thus to call explicitly
one of these functions.</p>
<p>We call them <em>lightweight</em> because we’ve taken care of making very few
changes to the language to preserve the semantics of existing
contracts: an entrypoint is simply an annotation, in the parameter
type of the contract, on the branch of interest in a tree of <code>or</code> types.</p>
<p>A consequence of this choice is that it is <strong>already possible today to
deploy a contract in Athens that would take advantage of entrypoints</strong>
if and when they get deployed.</p>
<p>When calling a contract’s entrypoint, instead of specifying the
succession of <code>Left</code>/<code>Right</code> constructors leading to the expected case,
we only need to explicitly specify the name of the entrypoint.</p>
<p>It brings more than just a convenience to avoid writing <code>Left</code>s and
<code>Right</code>s, as it allows one to call the entrypoint without any knowledge of
the contract’s full type — i.e. the type of all other entrypoints —
whereas today’s Michelson semantics forces any caller of a contract to
know it. Thus we gain flexibility in the interactions with and
especially <em>between</em> contracts.</p>
<h2 id="current-michelson-semantics"><a class="toclink" href="#current-michelson-semantics">Current Michelson semantics</a></h2>
<p>In all pasts and current protocols (including Athens), there is no
explicit entrypoint. The usual solution to this problem is to encode
entrypoints thanks to the type disjunction <code>or</code>. As an example, consider
the parameter type of a contract having multiple functionalities: spending the
token of the contract, setting its delegate, doing some business requiring a string as input, and
getting a callback answer carrying a <code>string</code> and a <code>nat</code>.</p>
<div class="highlight"><pre><span></span><code><span class="k">parameter</span>
<span class="w"> </span><span class="p">(</span><span class="kt">or</span>
<span class="w"> </span><span class="p">(</span><span class="kt">or</span><span class="w"> </span><span class="k k-Name">%manage</span>
<span class="w"> </span><span class="p">(</span><span class="kt">or</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="k k-Name">%spend</span><span class="w"> </span><span class="kt">mutez</span><span class="w"> </span><span class="kt">key_hash</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">key_hash</span><span class="w"> </span><span class="k k-Name">%set_delegate</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">unit</span><span class="w"> </span><span class="k k-Name">%default</span><span class="p">))</span>
<span class="w"> </span><span class="p">(</span><span class="kt">or</span>
<span class="w"> </span><span class="p">(</span><span class="kt">string</span><span class="w"> </span><span class="k k-Name">%contract_business</span><span class="p">)</span>
<span class="w"> </span><span class="p">(</span><span class="kt">pair</span><span class="w"> </span><span class="k k-Name">%contract_callback</span><span class="w"> </span><span class="p">(</span><span class="kt">string</span><span class="w"> </span><span class="k k-Name">%id</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">%value</span><span class="p">))));</span>
</code></pre></div>
<p>Any contract wanting to perform a call to e.g. the last entrypoint needs
to wrap its parameter (of type <code>(pair string nat)</code>) in a sequence of two <code>Right</code>s.
This is not satisfactory for at least two reasons:</p>
<ul>
<li>the position of the entrypoint of the target contract
leaks into the code of the caller, which is just plain ugly</li>
<li>another target contract with a similarly typed entrypoint but at a
different position would require a different wrapping in the caller</li>
</ul>
<p>Clearly, this solution doesn’t scale and is not composable.
To bypass these issues, one would have to rely on coincidental
polymorphism offered by <code>PACK</code>/<code>UNPACK</code>, or to originate proxy contracts
satisfying other contract’s expectations while carrying
information in their storage to proceed as intermediaries between our
contract and third-party contracts. Not very satisfactory either.</p>
<h2 id="naming-entrypoints-of-a-contract-and-calling-an-entrypoint"><a class="toclink" href="#naming-entrypoints-of-a-contract-and-calling-an-entrypoint">Naming entrypoints of a contract and calling an entrypoint</a></h2>
<p>Having multiple named entrypoints brings more polymorphism, hiding
irrelevant parts of the parameter type to callers: the third-party
can use the entrypoint name to call our contract and it just needs
to know the type of the data expected by this entrypoint - neither
the path that leads to this entrypoint, nor the type of
other branches have to be known. In our previous example, using the
new functionality, a third party contract can send data to our
entrypoint using <code>CONTRACT %contract_callback (pair string nat)</code>.</p>
<p>Here are the rules to define entrypoints. First of all, entrypoints
need to be named in the contract’s parameter type. To do so, the parameter
has to be a tree of <code>or</code> types (with possibly other types at the leaves).
Any annotation of a type constructor in this nesting of <code>or</code> types accessible
through <code>Left</code>/<code>Right</code> will become an entrypoint.
So, in the example above <code>%manage</code>, <code>%spend</code>, <code>%set_delegate</code>,
<code>%contract_business</code>, <code>%contract_callback</code> are entrypoints, while <code>%id</code>
and <code>%value</code> are not. Note that it is possible for entrypoints
to be nested: <code>%spend</code> and <code>%set_delegate</code> are under the <code>%manage</code> point.</p>
<p>The only constraints are, first, that a contract cannot define two entrypoints
with the same name and, second, that, should there be an entrypoint named <code>%default</code>
then all branches of the disjunctive type <strong>must</strong> be annotated with some
entrypoint. Otherwise, we could have reachability problems: consider
the (wrong) parameter type:</p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="kt">or</span><span class="w"> </span><span class="p">(</span><span class="kt">or</span><span class="w"> </span><span class="k k-Name">%default</span><span class="w"> </span><span class="p">(</span><span class="kt">mutez</span><span class="w"> </span><span class="k k-Name">%amount</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="kt">nat</span><span class="w"> </span><span class="k k-Name">%counter</span><span class="p">)</span><span class="w"> </span><span class="kt">string</span><span class="p">))</span>
</code></pre></div>
<p>It is impossible to access the branch of type <code>string</code>: calling
the contract without entrypoint will select <code>%default</code>.</p>
<h2 id="what-will-happen-to-my-old-contract"><a class="toclink" href="#what-will-happen-to-my-old-contract"><span class="dquo">“</span>What will happen to my old contract?”</a></h2>
<p>The overwhelming majority of contracts with scripts will stay completely untouched.
A few contracts originated before this amendment may undergo
automated transformations to render them compatible with the new
protocol. All these transformations are done in a way which minimizes
breakage and mostly preserves the semantics.
See <a href="#deprecation/whathappens">below</a> for more information.</p>
<h1 id="more-flexible-big-maps"><a class="toclink" href="#more-flexible-big-maps">More flexible big maps</a></h1>
<p>Relaxing the constraints on big maps is the other big quality-of-life
improvement we bring to Michelson. Big maps are used for storing anything
that does not need to be loaded upfront when calling a smart contract.
Having only one big map led to unhealthy development patterns. This
proposal fixes that issue.</p>
<h2 id="overview"><a class="toclink" href="#overview">Overview</a></h2>
<p>Currently (in Athens), each contract can have at most one big map
in its storage. Our proposal lifts this restriction and allows contracts
to store and manipulate an arbitrary amount of big maps in their storage.
What’s more, contracts calls can take big maps as arguments! The instruction
<code>EMPTY_BIG_MAP k v</code> is introduced to push an empty big map with keys
of type <code>k</code> and values of type <code>v</code> on the stack.</p>
<p>Not all restrictions are lifted, though: big maps cannot be shared among
contracts, i.e. a big map can appear in the storage of at most one contract. Moreover:</p>
<ul>
<li>big map types cannot be nested or appear as arguments of <code>PUSH</code> and
<code>UNPACK</code> instructions</li>
<li>when duplicating a big map, the full storage cost for duplication is
charged (no sharing)</li>
</ul>
<h2 id="under-the-hood"><a class="toclink" href="#under-the-hood">Under the hood</a></h2>
<p>The machinery for handling big maps got slightly more complicated in
order to preserve the invariants mentioned above. Now, big maps are
not stored along contracts but rather in a dedicated area of the
protocol state. Contracts manipulate big maps by their internal ids,
which are internally represented as integers. More details will be
provided with the full changelog for this proposal.</p>
<h1 id="new-instructions"><a class="toclink" href="#new-instructions">New instructions</a></h1>
<h2 id="stack-manipulation-instructions"><a class="toclink" href="#stack-manipulation-instructions">Stack manipulation instructions</a></h2>
<p>The new stack manipulation instructions aim at making compilation from
higher-level languages easier, but they should also be very useful for
those savvy users that write directly in Michelson. Currently, the stack
is manipulated “one element at a time”: in order to access an element
deep down in the stack, one has to perform a complex sequence of
reshuffling instructions. The new instructions capture these typical
use cases. In what follows, we will denote by <code>'a{n}</code> families of
type variables indexed by the naturals. The behaviours of these instructions
is pretty straightforward and is easily deductible from their types.</p>
<ul>
<li>The <code>DIG n</code> instruction takes the element at depth <code>n</code> of the stack
and moves it on top. The element on top of the stack is at depth <code>0</code>
so that <code>DIG 0</code> is a no-op.
The type of <code>DIG n</code> is the following:</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="n">DIG</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">::</span><span class="w"> </span><span class="s">'a{1} : ... : '</span><span class="n">a</span><span class="p">{</span><span class="n">n</span><span class="p">}</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="s">'b : '</span><span class="n">C</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="s">'b : '</span><span class="n">a</span><span class="p">{</span><span class="mi">1</span><span class="p">}</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="p">...</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="s">'a{n} : '</span><span class="n">C</span>
</code></pre></div>
<ul>
<li><code>DUG n</code> does the opposite of <code>DIG n</code>. It places the element on top
of the stack at depth <code>n</code>. It follows that <code>DUG 0</code> is also a no-op.
Here is its type:</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="n">DUG</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">::</span><span class="w"> </span><span class="s">'b : '</span><span class="n">a</span><span class="p">{</span><span class="mi">1</span><span class="p">}</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="p">...</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="s">'a{n} : '</span><span class="n">C</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="s">'a{1} : ... : '</span><span class="n">a</span><span class="p">{</span><span class="n">n</span><span class="p">}</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="s">'b : '</span><span class="n">C</span>
</code></pre></div>
<ul>
<li>The new <code>DIP n { code }</code> generalizes <code>DIP { code }</code> and allows to
execute <code>code</code> protecting the <code>n</code> first elements of the stack.
<code>DIP 1 { code }</code> is equivalent to the vanilla <code>DIP { code }</code> and
<code>DIP 0 { code }</code> is equivalent to executing <code>code</code>.
Assuming <code>code :: 'C -> 'D</code>, one has</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="n">DIP</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">::</span><span class="w"> </span><span class="s">'a{1} : ... : '</span><span class="n">a</span><span class="p">{</span><span class="n">n</span><span class="p">}</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="s">'C -> '</span><span class="n">a</span><span class="p">{</span><span class="mi">1</span><span class="p">}</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="p">...</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="s">'a{n} : '</span><span class="n">D</span>
</code></pre></div>
<ul>
<li><code>DROP n</code> drops the <code>n</code> top elements of the stack in one go.
As can be expected, <code>DROP 0</code> is a no-op. One has:</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="n">DROP</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="o">::</span><span class="w"> </span><span class="s">'a{1} : ... : '</span><span class="n">a</span><span class="p">{</span><span class="n">n</span><span class="p">}</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="s">'C -> '</span><span class="n">C</span>
</code></pre></div>
<p>While these instructions can be implemented with existing opcodes,
native support allows their gas cost to be much lower. More <span class="caps">CPU</span> is
typically consummed passing control between two opcodes than is consummed
executing simple stack manipulation operations.</p>
<h2 id="partial-application"><a class="toclink" href="#partial-application">Partial application</a></h2>
<p>Another instruction making its way in Michelson is <code>APPLY</code> which
allows performing the partial application of an argument to a lambda.
To be more precise, <code>APPLY</code> performs one step of currying followed
by a partial application in one go. The type of <code>APPLY</code> is the following:</p>
<div class="highlight"><pre><span></span><code><span class="kr">APPLY</span><span class="w"> </span><span class="o">::</span><span class="w"> </span><span class="s">'a : lambda (pair '</span><span class="n">a</span><span class="w"> </span><span class="s">'b) '</span><span class="n">c</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="s">'S -> lambda '</span><span class="n">b</span><span class="w"> </span><span class="s">'c : '</span><span class="n">S</span>
</code></pre></div>
<p>and its semantics is as follows:</p>
<div class="highlight"><pre><span></span><code>APPLY / x : code : S => { PUSH 'a x ; PAIR ; code } : S
</code></pre></div>
<p>This is a functional programming idiom that was sorely missed by some
Michelson practitioners. It should help as well for code generation from
higher-level languages, such as SmartPy or <span class="caps">LIGO</span>.</p>
<h2 id="chain-id"><a class="toclink" href="#chain-id">Chain Id</a></h2>
<p>In the third stage of the voting procedure, a test chain is forked to
let stakeholders try the new proposed protocol. We presented this
test chain <a href="https://research-development.nomadic-labs.com/athens-on-the-testchain.html">when the Athens proposal reached the third
stage</a>.</p>
<p>A transaction that is run in the test chain cannot be replayed on the
main chain by an attacker thanks to the <em>chain identifier</em> (chain id
for short): each chain uses a different identifier that is signed in
each transaction.</p>
<p>Until now, smart contracts had no way to access the chain id of the chain they run onto and in particular they had to behave identically on the
test chain and on the main chain.</p>
<p>In our proposed amendment, a new <code>chain_id</code> type is added to represent
Tezos chain identifiers. The new <code>CHAIN_ID</code> instruction can be used to
produce a value of this type, the current chain identifier.</p>
<h1 id="comparable-types-closed-under-products"><a class="toclink" href="#comparable-types-closed-under-products">Comparable types closed under products</a></h1>
<p>In Michelson, collections such as <code>set</code>s, <code>map</code>s and <code>big_map</code>s
are indexed by elements which must be endowed with the structure
of a total order. A slang inherited from OCaml calls these types
“comparable”. Up to now, only finitely many types are comparable:
<code>string</code>, <code>int</code>, <code>key_hash</code>, etc
(look <a href="https://tezos.gitlab.io/whitedoc/michelson.html#core-data-types-and-notations">here</a>
and look for “comparable types” for a full list).</p>
<p>Alas, this prevents using structured data to index maps
(especially for <code>big_map</code>). A possible hack to circumvent this limitation
is to first serialise the keys to bytes using <code>PACK</code> and then use
the serialised key. This is both costly and dangerous, as it forgets
typing information.</p>
<p>The new amendment partially lifts this restriction and allows us to use
finite products of comparable types as keys in types <code>set</code> and <code>map</code>.
Concretely, we implicitly endow finite products with the lexicographical
order constructed on that of the underlying base types.</p>
<h1 id="deprecationssemantics-alterations"><a class="toclink" href="#deprecationssemantics-alterations">Deprecations/semantics alterations</a></h1>
<h2 id="overview-of-the-deprecations"><a class="toclink" href="#overview-of-the-deprecations">Overview of the deprecations</a></h2>
<p>The main deprecations are the following:</p>
<ul>
<li><code>STEPS_TO_QUOTA</code> is deprecated, as its semantics was dependent on a
gas cost model subject to constant evolution and as it made evaluation
of Michelson smart contracts non-monotonic in the quantity of gas provided.</li>
<li><code>CREATE_ACCOUNT</code> disappears.</li>
<li><code>CREATE_CONTRACT</code> changes type. In our proposal, its type becomes:</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="n">CREATE_CONTRACT</span><span class="w"> </span><span class="o">::</span><span class="w"> </span><span class="n">option</span><span class="w"> </span><span class="n">key_hash</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">mutez</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="s">'g : '</span><span class="n">S</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">operation</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">address</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="s">'S</span>
</code></pre></div>
<p>This reflects the disappearance of the <code>manager</code>, <code>spendable</code> and <code>delegatable</code> fields.
- <code>NOW</code> has altered semantics: it returns the timestamp of the previous block plus 60 seconds.
This partially mitigates selfish behaviour where a baker would selfishly arbitrate the timestamp
of the current block.</p>
<h2 id="what-happens-to-existing-contracts"><a class="toclink" href="#what-happens-to-existing-contracts"><a name="deprecation/whathappens"></a>What happens to existing contracts?</a></h2>
<p>At stitching time (if switching from <code>Athens</code> to this new proposal),
some contracts may be augmented with extra entrypoints to render them compatible with the new protocol.
The details of this process are partially described on
<a href="https://medium.com/cryptium/meanwhile-at-cryptium-labs-1-part-vi-540170f46c51">Cryptium Labs’ blog</a>
and will be detailed with the full changelog of the protocol.</p>
<p>Keeping it short (and incomplete), the fate of a pre-existing contract will
depend on its nature.</p>
<ul>
<li>Scriptless contracts will be upgraded (for free!) to smart contracts replicating
the functionality of the deprecated <code>manager</code>, <code>spendable</code> and <code>delegatable</code> fields
using the new entrypoint mechanism.
The precise code used will depend on the state of these fields at stitiching time
(so that non spendable contracts do not become spendable, for instance).</li>
<li>Spendable smart contracts will we augmented with an extra <code>%do</code>
entrypoint that will allow the tz1 who was its manager in Athens to
execute arbitrary code, in particular to set the delegate or spend
the funds. <strong>We will provide a detailed explanation of how to
migrate existing software</strong> that uses scriptless KT1s, and **the
command line client will implement backwards compatibility for
commands <code>transfer</code> and <code>set delegate</code>. The current parameter type
of these contracts will be assigned to the default entrypoint.</li>
<li>Delegatable smart contract will be augmented with <code>set_delegate</code> and
<code>remove_delegate</code> entrypoint in a similar fashion.</li>
<li>A corner case is the one of contracts that are deployed before the
migration, and have a <code>%default</code> entrypoint. Without a special
treatment, these contracts would change type after the migration to
be of the type of their default entrypoint. For this, they are
augmented with a <code>%root</code> entrypoint at the root, and a special
treatment is done for them, so that their <code>%default</code> and <code>%root</code>
entrypoints can be considered interchangeable where needed.</li>
</ul>
<p>We are doing the final tests to ensure that all contracts deployed as
of today will be fully operational after the migration.</p>
<p>The only case of possible breakage is with <code>CREATE_CONTRACT</code>. Indeed,
this instruction may succeed to produce a transaction, that in turn
may fail if the code contains deprecated instructions. The rationale
for that is trivial: all code already introduced in the blockchain can
use deprecated instructions, but no new code can be introduced that
includes deprecated instructions. This is currently a non-issue, as no
contract uses this instruction in the chain (there are few use cases
for it).</p>
<h2 id="how-should-i-prepare-as-a-contract-author"><a class="toclink" href="#how-should-i-prepare-as-a-contract-author">How should I prepare as a contract author?</a></h2>
<p>For contract writers who are currently developping smart contracts
that are to be deployed both before of after the migration, we
recommend the following practices.</p>
<p>First, in the unlikely scenario where you were using some of the instructions
being deprecated, please do not use them. If the migration happens, you will have
to rewrite your contract if you want to redeploy it later.</p>
<p>Regarding entrypoint annotations, either make sure they are well
formed, or do not use them at all.</p>
<p>Related to the previous points: <strong>try deploying your smart contract in
a sandbox with the new protocol</strong>.</p>
<p>Finally, avoid using <code>CREATE_CONTRACT</code> if possible. This instruction
is a tricky one, tied with the semantics of the chain, and it may be
upgraded again in a future update. A simple pattern to originate
contracts from a contract is to wrap the <code>CREATE_CONTRACT</code> in a
lambda, and have the contract accept an authenticated <code>(lambda unit
(list operation))</code> exactly as the <code>manager.tz</code> contract does.</p>
<h1 id="gas-cost-overhaul"><a class="toclink" href="#gas-cost-overhaul">Gas cost overhaul</a></h1>
<h2 id="foreword"><a class="toclink" href="#foreword">Foreword</a></h2>
<p>Gas costs are subject to change, as we continuously refine our understanding
of the resource consumption of the protocol. Never rely on hard-coded gas
constants and <code>--dry-run</code> your transactions. The following are important points
for this particular proposal.</p>
<ul>
<li>we do <strong>not</strong> change per-block and per-operation gas limits
(more on this below);</li>
<li>the update to Michelson gas costs does <strong>not</strong> impact the gas cost for vanilla
transactions (<code>tz1</code> to <code>tz1</code>). Other parts of this proposal could however
alter these costs slightly. We won’t be concerned with this here.</li>
</ul>
<h2 id="overview-of-the-gas-update"><a class="toclink" href="#overview-of-the-gas-update">Overview of the gas update</a></h2>
<p>Gas cost is a proxy for resource consumption, especially time consumption.
If you read this fine <a href="https://research-development.nomadic-labs.com/analysis-of-emmy.html">analysis of Emmy+</a>,
you know that ensuring timely termination of computations is paramount to
the health of the network.</p>
<p>To summarize, <strong>overestimating gas costs prevents developers from
writing more interesting contracts, while underestimating these costs leads
to possible attacks.</strong> It is important to get this right.</p>
<p>During the last six month, we spent a fair amount of energy building
a comprehensive infrastructure for benchmarking parts of the protocol
and fitting statistical models of their resource consumption to
data generated by benchmarking. We particularly focused on the Michelson
interpreter, as having expressive smart contracts is a key element in
the value proposition of Tezos.</p>
<p>The outcome of our work is new gas costs for Michelson instructions. Before we get to
the details, the following should be noted:
- our new proposal does not change per-block and per-operation gas limits
(more on this below);
- the update to Michelson gas costs does not impact the gas cost for vanilla
transactions (<code>tz1</code> to <code>tz1</code>). Other parts of the amendment could however
alter these costs slightly. We won’t be concerned with this here.</p>
<h2 id="methodology"><a class="toclink" href="#methodology">Methodology</a></h2>
<p>The goal is to obtain a predictive model of the execution time of
smart contracts as a function of the size of their initial data.
Once such a model is obtained, it can be mapped to the gas cost model which
is part of the protocol and which bills every executed instruction.</p>
<p>We opted for linear cost models, where the predicted execution
time of each instruction is decomposed into a fixed cost and an
operand-dependent cost. Here, “linear” means that the model is
going to be a linear combination of fixed, typically nonlinear
basis functions. All the difficulty lies in guessing these
nonlinearities by staring at the code and at the empirical data.</p>
<p>As an example, let’s take the <code>ADD</code> instruction on integers. Under the
hood, this instruction is dispatched to an efficient arbitrary-precision
arithmetic library. Generating a bunch of random stacks of integers up to 100 kb
and measuring execution time of the interpreter on <code>ADD</code>, we get the following
bunch of points:
<img src="images/benchmarks_gas/empirical.png" style="display:block;margin:auto;width:50%;" alt="execution time of ADD">
Here, the <span class="math">\(x\)</span> and <span class="math">\(y\)</span> axes correspond to the sizes, in bytes, of the pair of integers being
added while the <span class="math">\(z\)</span> axis measures execution time in nanoseconds. We observe
that the data is mostly lying on two hyperplanes symmetric wrt the <span class="math">\(x=y\)</span> plane.
We guess that the execution time of this instruction on integers <span class="math">\(x, y\)</span> of sizes
<span class="math">\(|x|, |y|\)</span> is of the form:</p>
<p><center><span class="math">\(\theta_0 + \theta_1 max(|x|, |y|)\)</span>.</center></p>
<p>Here, <span class="math">\(\theta_0, \theta_1\)</span> are unknown parameters (measured in seconds)
that have to be <em>fitted</em> to the empirical data. The conclusion of this
process is either the that the model is wrong or a set of parameters which
make the model effectively predictive. In this favourable example, our
guess is particularly good: when projecting the empirical data into
the basis spanned by our chosen nonlinearity (the <span class="math">\(max\)</span> function), we
see that the data is indeed living close to a linear subspace, easily
fitted (in red):
<img src="images/benchmarks_gas/validation.png" style="display:block;margin:auto;width:50%;" alt="fit"></p>
<p>The last step is to convert this model into a cost function labelled
in <em>gas</em>, implemented in the protocol. Our framework allows to perform
all these tasks mostly automatically (except for guessing a good linear model).
This is important, as there are close to one hundred instructions, not all as
easy to fit as addition. Automatisation lessens the risk of mistakes
being made in this rather tedious process.</p>
<h2 id="outcome"><a class="toclink" href="#outcome">Outcome</a></h2>
<p>As a result, we now have a reasonably tight cost model for 90% of
Michelson instructions. The missing ones are those involved with serialisation
and typechecking, which require deeper work.</p>
<p>Let’s have a look at the gas costs of some smart contracts with <span class="amp">&</span>
without the new features. These are tests you can easily replicate at home.
We’ll be comparing gas costs in zeronet, which runs
a preliminary version of the new protocol, to gas costs in a mainnet
sandbox. We’ll call these versions respectively <code>005</code> and <code>004</code>.
We’ll start with a simple loop:</p>
<div class="highlight"><pre><span></span><code><span class="k">parameter</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="p">;</span>
<span class="k">storage</span><span class="w"> </span><span class="kt">unit</span><span class="w"> </span><span class="p">;</span>
<span class="k">code</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">UNPAIR</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">DIP</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">DROP</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">DUP</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">NEQ</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">LOOP</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">int</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">SWAP</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">SUB</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">DUP</span><span class="w"> </span><span class="p">;</span><span class="w"> </span><span class="kr">NEQ</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">DROP</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">PUSH</span><span class="w"> </span><span class="kt">unit</span><span class="w"> </span><span class="kc">Unit</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">NIL</span><span class="w"> </span><span class="kt">operation</span><span class="w"> </span><span class="p">;</span>
<span class="w"> </span><span class="kr">PAIR</span>
<span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>This contract expects an integer specifying a number of iterations.</p>
<p>Originating this contract costs roughly the same in <code>004</code> and <code>005</code>:
13035 vs 12756. Each origination or contract call has a minimal fixed cost
of 10000 gas, for issuing the operation. In the case of an origination,
one also pays for typechecking and storage of the script.</p>
<p>Calling this contract on eg arguments 5, 5000 and 13000 in <code>004</code> costs
respectively 13197, 283530 and 769957 gas units. One will be
hard-pressed to go further: the per-operation gas limit is fixed to
800000 gas units both in <code>004</code> and <code>005</code>.</p>
<p>Running the contract with the same inputs in <code>005</code> yields much
lower costs: respectively 12878, 31687 and 61812 gas units.
In fact, one can push the iteration number up to roughly
200000, for which the gas cost is equal to 765984, close to the
gas limit. We observe that gas costs for the involved instructions
significantly decreases in <code>005</code>. Control flow and arithmetic should
be way cheaper that in <code>004</code>.</p>
<p>Another interesting example is that of token standards such as
the one announced
<a href="https://medium.com/tqtezos/implementing-asset-contracts-on-tezos-b74c8c6ecdc">here</a>.
A prototype implementation of this token standard is available
<a href="https://gitlab.com/tzip/tzip/blob/master/assets/FA1.2/ManagedLedger.tz">here</a>.</p>
<p>Looking at the large quantity of stack manipulation and control-flow related
instructions, we should have reason to expect significant savings in terms of
gas consumption for these important contracts. Origination costs are
<code>146289</code> units of gas in <code>004</code> and <code>145974</code> in <code>005</code>, a slight improvement.
More importantly, calling the two basic entrypoints <code>%mint</code> and <code>%burn</code>
have similar costs—even slightly higher in the case of <code>%burn</code>!</p>
<p>This is a consequence of the fact that runtime cost of control-flow and arithmetic
instructions is negligible compared to the costs of serialisation, typechecking and
storage incurred by manipulating big maps.</p>
<!-- TODO: bench crypto opts -->
<h2 id="whats-next-for-gas"><a class="toclink" href="#whats-next-for-gas">What’s next for gas</a></h2>
<p>We now have a robust cost model for a large subset of instructions,
and an infrastructure to continue updating the gas costs as the protocol
evolves. As said in the beginning, defining a proper cost model for serialisation,
typechecking and storage is going to be one of the top priorities of the gas
team for the rest of this year. We’ve more plans in mind to rethink the
architecture of ressource accounting in Tezos and make this part of the protocol
more transparent for users and bakers alike.</p>
<h1 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h1>
<p>This protocol update contains a wide array of new features which we think greatly improve
the usability of Michelson both as a programming language for power users
and as a target for emerging higher-level languages.</p>
<p>It is quite exciting to have the opportunity to continuously refine the protocol
thanks to Tezos’ self-amending nature, and we hope that the community will be as
enthusiastic when discussing and debating the content of this proposal.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Analysis of Emmy+2019-07-22T18:00:00+02:002019-07-22T18:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2019-07-22:/analysis-of-emmy.html<p><strong> 04/6/2020 update</strong>: After a discussion with Michael Neuder (University of Colorado Boulder), who noticed a discrepancy between his analysis of Emmy+ and ours, we identified a bug in the script used to generate the data for the plot in the “Forks starting now” section. We have therefore updated the plot. The number of confirmations has changed from <code>6</code>, <code>12</code>, <code>44</code> to <code>7</code>, <code>16</code>, <code>67</code> for attacker stake fractions of <code>0.2</code>, <code>0.3</code>, <code>0.4</code>, respectively. The discrepancy was due to taking into account the rate of the <em>shortest</em> adversarial forks that are faster than the honest chain; instead, we should have taken into account the rate of <em>any</em> fork faster than the honest chain. In addition, we have provided a more precise explanation of the last plot from the “Forks started in the past” section.</p>
<p><strong> 19/11/2019 update</strong>: Following feedback from Daniel Moroz and Michael Neuder, we edited the blog post in order to nuance our previous statement that “selfish baking is not profitable”. More precisely, selfish baking only gives tiny profits, for instance 0.0016% gains with respect to the honest rewards, for a dishonest baker with 30% stake fraction. A new table gives the details.</p>
<p><strong>Note</strong>: <em>This analysis was done with the help of Arthur Breitman and Bruno Blanchet (Inria). The code used for the analysis can be found at <a href="https://gitlab.com/eugenz/emmyplus">this url</a>.</em></p>
<hr>
<p>We have recently announced <a href="https://research-development.nomadic-labs.com/emmy-an-improved-consensus-algorithm.html">Emmy<sup>+</sup></a> an improvement of Emmy, the current consensus algorithm used by Tezos. In this blog post, we perform a brief analysis of Emmy<sup>+</sup>.</p>
<p><strong>Disclaimer</strong>: We emphasize that this is not a complete analysis; in particular, we do not present any security proofs. Also, the results of the analysis should not be extrapolated, they only hold inside the model we’re working with. Namely, we assume here that the network is synchronous and therefore all blocks and operations (including endorsements) are effectively broadcast to all participants within a known time bound, namely 40 seconds. Furthermore, we only consider a rather limited attacker, who needs not follow the protocol, but who is otherwise not capable of disturbing the network (by blocking or delaying messages between honest participants).</p>
<p>Let’s first recall the main ideas behind Emmy<sup>+</sup>: the number of endorsements included in a block no longer influences the fitness, which only increases by 1 with each level. Instead, the number of endorsements determines the minimal time at which a block is valid: the fewer endorsements a block carries the longer it takes before it can be considered valid. Furthermore, the reward policy is adjusted so it discourages selfish baking. As we will see, these changes increase the security of Tezos’ consensus layer.</p>
<p>We summarize next the main findings, which are obtained for the following security criterion: the rate at which a malicious fork can occur should be lower than 10<sup>-8</sup>, that is, roughly once every two centuries.</p>
<p>Assuming an attacker that has at most 33% of the total stake, in our model we obtain that an operation included 6 blocks ago will only end up on the attacker’s chain (which might exclude that operation) once every two centuries, given that the chain is perfectly healthy during these last 6 levels (that is, no skipped priorities, no missed endorsements).
Moreover, our analysis suggests that, for a freshly injected operation, and making no assumption on the health of the chain, <code>6</code>, <code>12</code>, and <code>44</code> block confirmations are recommended, assuming an attacker with a stake of <code>20%</code>, <code>30%</code>, and respectively <code>40%</code> of the total stake. Here again, the statement is not absolute, but instead relative to our security criterion.</p>
<p>Our analysis also shows that, in our model of Emmy<sup>+</sup>, selfish baking results in insignificant profits, even when the baker attempting it has a very large portion of the stake.</p>
<p>This blog post is structured in four parts:</p>
<ul>
<li>a presentation of the changes introduced by Emmy<sup>+</sup></li>
<li>an analysis of malicious forks</li>
<li>an analysis of selfish baking</li>
<li>a brief presentation of the rationale for choosing the values of Emmy<sup>+</sup>-specific constants</li>
</ul>
<h2 id="the-changes-introduced-in-emmy"><a class="toclink" href="#the-changes-introduced-in-emmy">The changes introduced in Emmy<sup>+</sup></a></h2>
<p>The improvements brought by Emmy<sup>+</sup> are achieved by means of a new “minimum block delay” function and adjustments to the reward policy.</p>
<p>The new delay function is defined as follows:</p>
<div class="highlight"><pre><span></span><code>emmy_plus_delay(p, e) =
time_between_blocks[0] + p * time_between_blocks[1] +
delay_per_missing_endorsement * max(0, (initial_endorsers - e))
</code></pre></div>
<p>In what follows we use the following abbreviations:</p>
<div class="highlight"><pre><span></span><code><span class="nv">dp</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nv">time_between_blocks</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="nv">de</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nv">delay_per_missing_endorsement</span>
<span class="nv">ie</span><span class="w"> </span><span class="o">:=</span><span class="w"> </span><span class="nv">initial_endorsers</span>
</code></pre></div>
<p>where <code>dp</code> stands for “delay per priority” (that is, per missed baking slot) and <code>de</code> for “delay per (missed) endorsement (slot)”. The chosen values are <code>dp = 40</code>, <code>de = 8</code>, and <code>ie = 24</code>.</p>
<p>In Emmy, the baking reward was set to <code>16</code> tez and the endorsing reward to <code>2/(p+1)</code> tez (per endorsement slot), where <code>p</code> is the priority of the endorsed block.</p>
<p>In Emmy<sup>+</sup>, the baking reward is <code>16/(p+1) * (0.8 + 0.2 * e/32)</code> and the endorsing reward is <code>2/(p+1)</code>, where <code>p</code> is the baked block’s priority or the priority of the block which contains the endorsement, respectively, and <code>e</code> is the number of included endorsements.
These changes counter selfish baking, as well as incentivize bakers to include all available endorsements.</p>
<p>Indeed, smaller rewards for smaller priorities make selfish baking attempts much less profitable: a baker with a slot at priority 1 who might attempt to steal the block at priority 0 by withholding his endorsements would be better off having his endorsements included in the block of priority 0. Note that the reward for his baked block does not have much weight, it is worth less than 4 endorsements on the block at priority 0.</p>
<p>Also, the factor <code>e/32</code> in the baking reward function is used to incentivize bakers to include all available endorsements. On the other hand, this factor only counts for 20% of the main factor of the baking reward (namely <code>16/(p+1)</code>) so that the risk for losing the baking slot outweighs the benefit of waiting for more endorsements.</p>
<h2 id="preliminaries"><a class="toclink" href="#preliminaries">Preliminaries</a></h2>
<!--- an attacker is a she, a baker is a he (based on whether the word starts with a vowel) -->
<p>To get an intuition of the event rates that we see in this post, we recall the following rough correspondences.</p>
<table>
<thead>
<tr>
<th>Rate (in blocks<sup>-1</sup>)</th>
<th align="right">Frequency</th>
</tr>
</thead>
<tbody>
<tr>
<td>10<sup>-3</sup></td>
<td align="right">10<sup>3</sup> minutes = 16.6 hours</td>
</tr>
<tr>
<td>10<sup>-4</sup></td>
<td align="right">6.9 days</td>
</tr>
<tr>
<td>10<sup>-5</sup></td>
<td align="right">2.2 months</td>
</tr>
<tr>
<td>10<sup>-6</sup></td>
<td align="right">1.9 years</td>
</tr>
<tr>
<td>10<sup>-7</sup></td>
<td align="right">1.9 decades</td>
</tr>
<tr>
<td>10<sup>-8</sup></td>
<td align="right">1.9 centuries</td>
</tr>
<tr>
<td>10<sup>-x</sup></td>
<td align="right">1.9 * 10<sup>(x-8)</sup> centuries</td>
</tr>
</tbody>
</table>
<p>Here we assumed that blocks are produced every 60 seconds. Consequently, given a rate of 10<sup>-x</sup> in blocks<sup>-1</sup> of an event <code>e</code>, <code>e</code> happens every 10<sup>x</sup> blocks, and therefore every 10<sup>x</sup> minutes.</p>
<p>Note that we talk about the <em>rate</em> at which an event happens rather than the <em>probability</em> of its occurrence, because, in contrast to a proof-of-work system like Bitcoin, if an attacker continuously attempts to make the event happen, she will for sure succeed (at a certain rate).</p>
<p>We also recall that 1mu = 10<sup>-6</sup>, 1n = 10<sup>-9</sup>, 1p = 10<sup>-12</sup>, and 1f = 10<sup>-15</sup>.</p>
<h2 id="malicious-forks"><a class="toclink" href="#malicious-forks">Malicious forks</a></h2>
<p>A malicious fork is one where an attacker is building an alternative branch in parallel with the main branch. The attacker can then make her branch become the main branch, if her branch is faster. Here the attacker’s goal is to break consensus. This is in contrast to the selfish baking strategies, where the goal of a dishonest baker is to obtain financial gain through the reward system. Therefore we assume that the attacker is not refraining from double baking and double endorsing.</p>
<h3 id="forks-started-in-the-past"><a class="toclink" href="#forks-started-in-the-past">Forks started in the past</a></h3>
<p>We are mainly interested in the following question: what is the rate at which an operation included in a block <code>n</code> levels ago will end up on an alternative branch? Here we assume that the attacker’s goal is to build an alternative branch in which the operation is not included. We therefore assume that the attacker starts building her branch at the same level as the block which contains the operation.</p>
<p>To answer this question, we consider the delay accumulated by the current chain, relative to the “ideal chain”, where all blocks are produced at priority 0 and have 32 endorsements. Note that the accumulated delay (called below “observed delay”) is the time difference between the timestamp of the current block and the timestamp of the block containing the operation, minus <code>n</code> minutes (corresponding to <code>n</code> observed blocks). We will see that this delay measures the “health” of the chain: the smaller the delay, the healthier the chain is. The operation ends up on the attacker’s branch if this branch becomes at some point in the future faster than the branch including the current block.</p>
<p>We plot the rates of such forks when the attacker’s stake fraction varies and for a few values of <code>n</code> and of the observed delay. Concretely, the different curves are for the cases when the number <code>n</code> of observed blocks is either <code>3</code> or <code>6</code> and when the observed delay is <code>0</code>, <code>128</code>, or <code>256</code> seconds. The observed delay can be expressed as <code>sp * dp + me * de</code>, where <code>sp</code> is the number of skipped priorities and <code>me</code> is the number of missed endorsements. Therefore an observed delay value of <code>128</code> may represent more than one possibility, like 0 skipped priorities and 16 missed endorsements, or 1 skipped priority and 11 missed endorsements, and so on (assuming <code>dp=40</code> and <code>de=8</code>).</p>
<p><img alt="Image" src="images/emmyplus/rate_forks_past_24_5_varying_n_delay_3_6.png"></p>
<p>These rates can be interpreted as follows. In our model, one can consider an operation included <code>n</code> blocks ago as final as long as:</p>
<ul>
<li><code>n=6</code> and there were no skipped priorities nor missing endorsements during the last 5 levels (that is, <code>delay=0</code>), assuming an attacker with at most 33% of the stake;</li>
<li><code>n=6</code> and, for instance, there are no skipped priorities, but around 2-3 missing endorsements per level, assuming an attacker with at most 31% of the stake;</li>
<li><code>n=6</code> and, for instance, 2 skipped priorities, but around 5 missing endorsements per level, assuming an attacker with at most 28% of the stake;</li>
<li><code>n=3</code> and again a perfectly healthy chain, assuming an attacker with at most 22% of the stake.</li>
</ul>
<!--
- `n=5` and there were no skipped priorities nor missing endorsements during the last 5 levels (that is, `delay=0`), assuming an attacker with at most 30% of the stake;
- `n=5` and, for instance, there are no skipped priorities, but around 3 missing endorsements per level, assuming an attacker with at most 27% of the stake;
- `n=5` and, for instance, 2 skipped priorities, but around 4-5 missing endorsements per level, assuming an attacker with at most 24% of the stake;
-->
<p>To get a better understanding of the case when the chain is not very healthy, we also plot the rate of forks as a function of <code>n</code> for 3 different stake fractions (<code>0.2</code>, <code>0.3</code>, <code>0.4</code>) and a delay proportional to the number <code>n</code> of observed blocks, namely <code>8*n*de</code>, representing that one quarter of endorsements are being missed per level (and no skipped priorities).</p>
<p><img alt="Image" src="images/emmyplus/rate_fork_fct_ob.png"></p>
<p>The plot shows that, in our model:</p>
<ul>
<li>for an attacker stake fraction of <code>0.2</code> (<code>0.3</code>, <code>0.4</code>),</li>
<li>for a number <code>n</code> of observed blocks at least <code>5</code> (<code>9</code>, <code>25</code>),</li>
<li>given that the observed delay is <code>8*n*de</code> seconds,</li>
</ul>
<p>the rate of having a fork of any length starting <code>n</code> blocks in the past is smaller than the one set by the security criterion.</p>
<h3 id="forks-starting-now"><a class="toclink" href="#forks-starting-now">Forks starting now</a></h3>
<p>Next, we look into the case where the malicious fork starts at the current block. Similarly to the previous case, we plot the rate of forks as a function of the fork length. This analysis suggests how many block confirmations are necessary, under our assumptions, no matter how unhealthy the chain will be.</p>
<p><img alt="Image" src="images/emmyplus/rate_fork_fct_fl.png"></p>
<p>The plot shows that, in our model, for our security criterion to be met, one needs to wait <code>7</code>, <code>16</code>, and <code>67</code> blocks, when assuming an attacker with a stake fraction of <code>0.2</code>, <code>0.3</code>, and respectively <code>0.4</code>.</p>
<h2 id="selfish-baking"><a class="toclink" href="#selfish-baking">Selfish baking</a></h2>
<p>With selfish baking, a dishonest baker tries to “steal” baking slots by building a faster alternative branch in order to gain more rewards. To achieve this, the baker’s only strategy, as far as we see, is to withhold his endorsements in order to slow down the honest branch.</p>
<p>We proceed as follows. We first present three selfish baking scenarios, which are among the most relevant ones. Next, we compute the rates at which these scenarios occur, and also the rate at which selfish baking in general is successful, in relation with the fraction of the stake of the dishonest baker. Finally, we compare the baker’s expected rewards on the honest branch with those on the dishonest branch, thus determining whether selfish baking is profitable.</p>
<h3 id="the-scenarios"><a class="toclink" href="#the-scenarios">The scenarios</a></h3>
<p>For the ease of reference, we denote the dishonest baker by D.
To illustrate the corresponding scenarios, consider the following tree with root block B<sub>0</sub>. The branch B<sub>1</sub>, B<sub>2</sub>, B<sub>3</sub> represents the chain being constructed by an honest baker, while B’<sub>1</sub>, B’<sub>2</sub>, B’<sub>3</sub> the one D tries to construct. Let t<sub>i</sub> (resp. t’<sub>i</sub>) be the time of block B<sub>i</sub> (resp. B’<sub>i</sub>). The label on top of a box represents the priority at which the block is created.</p>
<p><img alt="Image" src="images/emmyplus/attack_tree.png"></p>
<p>The three scenarios are as follows:</p>
<ol>
<li>D bakes his block B’<sub>1</sub> before B<sub>1</sub>, that is, t’<sub>1</sub> ≤ t<sub>1</sub>.</li>
<li>D bakes his block B’<sub>2</sub> before B<sub>2</sub>, that is, t’<sub>2</sub> ≤ t<sub>2</sub> (but t’<sub>1</sub> > t<sub>1</sub>).</li>
<li>D bakes his block B’<sub>3</sub> before B<sub>3</sub>, that is, t’<sub>3</sub> ≤ t<sub>3</sub> (but t’<sub>1</sub> > t<sub>1</sub> and t’<sub>2</sub> > t<sub>2</sub>).</li>
</ol>
<p>In scenarios (2) and (3) we assume that D also has the slots at priorities 1 to p<sub>2</sub> for the level corresponding to B<sub>2</sub>, and similarly, it also has the priorities 1 to p<sub>3</sub> for the level corresponding to B<sub>3</sub>. Therefore the lowest priority at which an honest baker can bake is p<sub>2</sub>+1 and respectively p<sub>3</sub>+1.</p>
<h3 id="block-stealing-rate"><a class="toclink" href="#block-stealing-rate">Block stealing rate</a></h3>
<p>We analytically compute the rates at which a baker could steal baking slots by withholding endorsements for the 3 different scenarios.
We also compute these rates in the general case, that is, for all possible selfish baking scenarios. More precisely, we consider all branches starting with a block at which the dishonest baker does not have priority 0. (If he has the best priority at some level, it is more profitable to simply bake the block at that level on the honest branch, instead of starting a fork.)</p>
<p>We plot the rate of forks for each of the three scenarios, the sum of these rates, and the rates for the general case, when the stake fraction varies. The graphic suggests that generalizing the scenarios to higher fork lengths (that is, higher than 3) is not worthwhile, because the rates become insignificant. The plots also shows that, at least when D does not have a very high stake, the 3 scenarios are the most relevant.</p>
<p><img src="images/emmyplus/scenarios.png" style="display:block;margin:auto;width:100%;" alt="scenarios"></p>
<p>We stress that here we do not take into account whether D would make a profit by selfish baking, but only whether D’s branch would be faster than the honest branch.
We will see next that, even when the rate at which D’s branch is faster is quite high, D cannot take advantage of this, as D’s (average) reward on the dishonest branch would not surpass the (average) reward on the honest branch.</p>
<h3 id="profitability-of-selfish-baking"><a class="toclink" href="#profitability-of-selfish-baking">Profitability of selfish baking</a></h3>
<p>The new reward policy ensures that selfish baking only gives insignificant profits, at least in our model.</p>
<p>Indeed, consider first scenario (1). Assume that D has <code>e</code> endorsement slots. If D plays correctly, sharing his endorsements, and if these are included in B<sub>1</sub>, then his reward is <code>2 * e</code>. If not, and assuming D is able to steal the baking slot, the reward for D is <code>16/(1+1)*(0.8+0.2*32/32) = 8</code> (assuming all 32 endorsements are available to D). Therefore, it is profitable for D to steal the block only when <code>8 > 2*e</code>, that is, <code>e < 4</code>. However, it is easy to see that, with <code>e < 4</code>, D cannot steal the baking slot: we have t<sub>1</sub> < t’<sub>1</sub>.</p>
<p>To further show that the new reward policy in Emmy<sup>+</sup> discourages selfish baking, we also compute the expected reward on the dishonest branch (under the condition that the dishonest branch is faster and is profitable), under both the Emmy<sup>+</sup> and the Emmy reward policies, assuming that we are in scenario (2).</p>
<p><img src="images/emmyplus/compare_expected_rewards.png" style="display:block;margin:auto;" alt="compare_expected_rewards"></p>
<p>We observe that, for instance, for a stake fraction of <code>0.2</code>, the expected reward with Emmy<sup>+</sup> rewards is about 6 times less than with Emmy rewards, and about 40 times less for a stake fraction of <code>0.4</code>.</p>
<p>Finally, we compute the expected rewards in the general case, that is, for all possible selfish baking scenarios. More precisely, we compute the expected reward that D would obtain on the honest branch, and the expected reward D would obtain on the dishonest branch when it is faster than the honest chain. We plot the percentage of the expected reward on the dishonest branch with respect to the honest branch. To put these values in perspective, we also plot again the rate at which the dishonest chain is faster (meaning that block stealing is possible, though not necessarily profitable).</p>
<p><img src="images/emmyplus/percentage_rewards_rate.png" style="display:block;margin:auto;" alt="percentage_rewards_rate"></p>
<p>We observe that the expected reward on the dishonest chain is always smaller than the expected reward on the honest chain. We also see that for very small stakes (that is, a stake fraction between <code>0.01</code> and <code>0.05</code>), the expected reward on the dishonest chain is higher than 80% of the expected reward on the dishonest chain. However, in our model, the rate at which such cases occur is less than <code>10p</code>, that is, less frequent as once per century. Intuitively, there is a peak for small stakes because, when the baker’s stake is small, an additional baked block makes a big difference in terms of rewards.
Note also that this analysis assumes that the dishonest baker does not know in advance whether he will make profits, therefore he always performs selfish baking.</p>
<p>Finally, the following table considers the profits the dishonest baker gains with forks of length 2. As discussed above, the baker does not obtain profits with forks of length 1, while forks of length higher than 2 have very low rates of feasibility. The table shows the rate (of success per block), the profits (in tez) per year, and the percentage of the gains with respect to the rewards expected to be obtained by honest baking.</p>
<table>
<thead>
<tr>
<th align="center">stake ratio</th>
<th>rate</th>
<th align="right">profits per year</th>
<th align="right">percentage gain</th>
</tr>
</thead>
<tbody>
<tr>
<td align="center">0.10</td>
<td>0.000000</td>
<td align="right">0.268869</td>
<td align="right">0.000006 %</td>
</tr>
<tr>
<td align="center">0.15</td>
<td>0.000007</td>
<td align="right">6.340287</td>
<td align="right">0.000101 %</td>
</tr>
<tr>
<td align="center">0.20</td>
<td>0.000061</td>
<td align="right">43.331616</td>
<td align="right">0.000515 %</td>
</tr>
<tr>
<td align="center">0.25</td>
<td>0.000235</td>
<td align="right">129.935813</td>
<td align="right">0.001236 %</td>
</tr>
<tr>
<td align="center">0.30</td>
<td>0.000499</td>
<td align="right">207.868605</td>
<td align="right">0.001648 %</td>
</tr>
<tr>
<td align="center">0.35</td>
<td>0.000650</td>
<td align="right">196.642652</td>
<td align="right">0.001336 %</td>
</tr>
<tr>
<td align="center">0.40</td>
<td>0.000586</td>
<td align="right">115.204479</td>
<td align="right">0.000685 %</td>
</tr>
<tr>
<td align="center">0.45</td>
<td>0.000414</td>
<td align="right">42.734604</td>
<td align="right">0.000226 %</td>
</tr>
</tbody>
</table>
<h2 id="choosing-the-constants"><a class="toclink" href="#choosing-the-constants">Choosing the constants</a></h2>
<h3 id="overview"><a class="toclink" href="#overview">Overview</a></h3>
<p>Two main goals guided us when choosing the value of the constants: minimizing the profitability of selfish baking and minimizing fork rates (equivalently, maximizing finality). As we will see, these two goals are conflicting for some of the constants.</p>
<p>Also, we note that for the kinds of analysis we have seen, it is not the concrete values of the <code>dp</code> and <code>de</code> constants that matter, but instead their ratio, <code>dp/de</code>. Indeed, for such analyses, we are interested in which branch of two given branches is faster, that is whether or not </p>
<div class="math">$$\sum_{i=1}^{k}delay(p_i,e_i)<\sum_{i=1}^{k'}delay(p_i',e_i')$$</div>
<p> holds, where <span class="math">\(k\)</span>, <span class="math">\(k'\)</span> are the lengths of the two branches, and <span class="math">\(p_i, e_i, p_i', e_i'\)</span> are the priorities and the number of endorsements of the blocks of the two branches, respectively. By expanding the <code>delay</code> expression, we obtain the following inequality
</p>
<div class="math">$$\frac{dp}{de}<\frac{\big(\sum_{i=1}^{k'}max(0,(ie - e_i'))\big) - \big(\sum_{i=1}^{k}max(0,(ie - e_i))\big)}{\big(\sum_{i=1}^{k}p_i\big) - \big(\sum_{i=1}^{k'}p_i'\big)}$$</div>
<p>
This inequality shows that which branch is faster depends on the <code>dp/de</code> ratio rather than the values of <code>dp</code> and <code>de</code> taken independently.</p>
<p>We have taken <code>dp</code> to be <code>40</code> as it seems a reasonable value: it reduces the waiting time between blocks (with respect to the value in Emmy, <code>75</code>), but it is still large enough to allow blocks and operations to be timely delivered.</p>
<h3 id="choosing-init_endorsements"><a class="toclink" href="#choosing-init_endorsements">Choosing <code>init_endorsements</code></a></h3>
<p>Intuitively, a lower <code>ie</code> value is better for preventing selfish baking and a higher value is better for preventing malicious forks.</p>
<p>Indeed, with a low <code>ie</code> a baker that wants to steal baking slots has less of a leverage by withholding endorsements. Say that the dishonest baker D has <code>e'</code> endorsements and say <code>32-e'</code> endorsements are available to the honest bakers. For D to be able to delay the block honest bakers are baking, we need that <code>ie - (32 - e') > 0</code>, that is <code>ie + e' > 32</code>. So the lower <code>ie</code> is, the higher the number <code>e'</code> of endorsements D needs to have.</p>
<p>Regarding forks, the higher the <code>ie</code>, the more weight we give to the number of endorsements on the chain. This results in fewer chances for an attacker to build a faster alternative chain, as the number of endorsements available to him is independent of <code>ie</code>.</p>
<p>Next, we check the above intuition.
We first consider the plot for the rates of of block stealing.</p>
<p><img alt="Image" src="images/emmyplus/rates_steal_ie_log.png"></p>
<p>The plot suggests 20 as the value minimizing the rates for each given stake fraction.</p>
<p>We now consider fork rates when the number <code>n</code> of observed blocks is <code>5</code> and the observed delay is <code>128</code>. We note that we obtain similar shapes for different <code>n</code> and <code>delay</code> values.</p>
<p><img alt="Image" src="images/emmyplus/rates_varying_ie_log.png"></p>
<p>This confirms that higher values for <code>ie</code> correspond to smaller probabilities of forks.</p>
<p>We choose <code>24</code> as a trade-off value between lower values to prevent selfish baking and higher values to reduce malicious forks. As we have seen, the new reward policy in Emmy<sup>+</sup> reduces even further the profitability of selfish baking.</p>
<h3 id="choosing-dpde"><a class="toclink" href="#choosing-dpde">Choosing <code>dp/de</code></a></h3>
<p>For the chosen value of <code>ie</code>, that is <code>24</code>, we now vary the <code>dp/de</code> ratio.
We obtain the following graph for the rates of block stealing.</p>
<p><img alt="Image" src="images/emmyplus/rates_steal_dpde_log.png"></p>
<p>Here we see that a ratio of roughly <code>7</code> minimizes all cases.</p>
<p>Next, we look at fork rates. As above, the plots are for <code>n = 5</code> and <code>delay = 128</code>.</p>
<p><img alt="Image" src="images/emmyplus/rate_forks_dpde5_n_5_delay_128_log.png"></p>
<p>Again, we choose <code>dp/de = 5</code> as a trade-off value that is good enough for both our goals. This leads to the value <code>de = 8</code>.</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>An indexer for Tezos2019-07-16T18:00:00+02:002019-07-16T18:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2019-07-16:/an-indexer-for-tezos.html<p>We are happy to announce that the <em>indexer for Tezos</em> we have been
working on is ready for beta-testing and available
<a href="https://gitlab.com/nomadic-labs/tezos-indexer">here</a>.</p>
<p>But first…</p>
<h2 id="what-is-an-indexer-and-what-is-it-useful-for"><a class="toclink" href="#what-is-an-indexer-and-what-is-it-useful-for">What is an indexer, and what is it useful for?</a></h2>
<p>Mainly, indexers fill a void by providing information that’s not
directly available from the node’s <span class="caps">RPC</span> interface. But then why
don’t nodes provide these RPCs in the first place? That’s simply
because when you design and implement your node, you want to provide
just what is necessary, focus on what’s most important, in order to
reduce the possible sources of errors or bugs or security
issues. Also, it’s not really a node’s job to provide all
information that users can just deduce or derive from what it does provide.</p>
<p>However, some information just cannot be rapidly deduced from the
combination of a reasonable number of RPCs to the node. For instance,
if you want the ten latest operations done by a baker, you don’t want
to scan the list of transactions until you find ten of them,
especially if there’s fewer than ten, since in that case it would make
you go up to the genesis block! Instead, you want to have an <strong>index</strong>
of all the transactions done by that baker in a database that’s both
easy to access and fast.</p>
<p>Wallets, DApps (decentralized applications, often blockchain-enabled
websites), and block explorers are the main users of indexers.</p>
<h2 id="some-technical-details-about-our-indexer-for-tezos"><a class="toclink" href="#some-technical-details-about-our-indexer-for-tezos">Some technical details about our indexer for Tezos</a></h2>
<p>The code of the indexer for Tezos we are presenting here is a
refactoring of Mezos, which is the middleware used by the wallets
developed by the Nomadic Labs mobile team. It already has been
used extensively and developed in the last 12 months, and its design is
fairly stable.</p>
<p>However there have been quite a few changes, going from Mezos to this
indexer and here are some of them:</p>
<ul>
<li>For bootstrapping, instead of having to run 4+3=7 slightly different
versions of Mezos, you need to run just a single version of this
indexer: it handles the 4 different protocols and the 3 protocol
transition blocks. That applies to Mainnet only. Other networks are
not supported at all for the time being (it wouldn’t be difficult,
it just hasn’t been given a priority yet).</li>
<li>Mezos uses TzScan to get the snapshot blocks. However this indexer
doesn’t use TzScan at all, it directly implements that part so we
could break free from some closed source code.</li>
</ul>
<h2 id="location"><a class="toclink" href="#location">Location</a></h2>
<p>You can find it
<a href="https://gitlab.com/nomadic-labs/tezos-indexer">here</a>,
and if you want to play with it, start by reading the
<a href="https://gitlab.com/nomadic-labs/tezos-indexer/-/blob/master/README.md"><span class="caps">README</span></a>.</p>
<p>Have fun! :)</p>Meanwhile at Nomadic Labs #32019-07-08T18:00:00+02:002019-07-08T18:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2019-07-08:/meanwhile-at-nomadic-labs-3.html<p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>Another update on the many projects we have been busy working on.
Most of the following topics will be subject of more in-depth posts of
their own.</p>
<h1 id="protocol"><a class="toclink" href="#protocol">Protocol</a></h1>
<p>The consensus team is finishing their analysis of selfish baking on
Emmy+ and tweaking the constants of the protocol accordingly.
At the same time work on our version of Tendermint is progressing to
the point of having a proof of concept protocol to play with.</p>
<p>The protocol team has been steadily reviewing and testing all the MRs
that are flowing in the branch
<a href="https://gitlab.com/tezos/tezos/-/merge_requests?state=merged&target_branch=proto-proposal">proto-proposal</a>.
Particularly the excellent work of <a href="https://medium.com/cryptium">Cryptium
Labs</a> on the account re-haul (implicit
accounts, removing script-less contracts) and the tweaks to the voting
procedure (quorum caps, proposal quorum).</p>
<p>The Michelson team has been busy too with quite a lot of changes that
we’ll blog about soon. For a quick sneak peek:</p>
<ul>
<li>Smart contracts will have multiple entrypoints, implemented in a way
that is not too intrusive and that is retrocompatible (provided that
they conform to the entrypoint convention, contracts that get
deployed now will take advantage of multiple entrypoints if and when
the update is rolled).</li>
<li>Most restrictions on the <code>big_map</code> type got lifted.</li>
<li>New instructions have been introduced to help large contracts (in
particular compiled from higher level languages) stay compact and
efficient: new deep stack access instructions and an instruction to
partially apply a <code>lambda</code>.</li>
<li>The gas cost model for instructions has been refined, based on benchmarks.</li>
<li>Some instructions have been updated or deprecated according to the
account re-haul.</li>
</ul>
<h1 id="shell"><a class="toclink" href="#shell">Shell</a></h1>
<p>The shell is looking at an interesting summer update with a new
storage back-end from <a href="https://tarides.com/">Tarides</a>, that is
currently benching and fine tuning the performance of their code.</p>
<p>The mempool team is slowly but steadily merging the huge amount of
work that they have been doing in the last 6 months.
Most of the effort is being devoted to testing the new mempool against
spam attacks which is a rather complex task.</p>
<p>Finally there are small improvements to the p2p layer that will
improve connectivity, network topology and peer discovery.</p>
<h1 id="tooling"><a class="toclink" href="#tooling">Tooling</a></h1>
<p>We are close to the release of our own <code>tezos-indexer</code>, an improved
iteration over the previous project Mezos, which will serve as base
for our mobile wallets, Cortez for
<a href="https://play.google.com/store/apps/details?id=com.tezcore.cortez&hl=en">Android</a>
and <a href="https://apps.apple.com/us/app/cortez/id1464922586">iOS</a>, and many
future projects.</p>
<p>The first project from one of our interns is almost ready for release,
Try Michelson.
You can now run a Michelson contract in a friendly web interface
without needing to run a node.
Try Michelson is already proving to be a great tool to teach newcomers
how to write smart contracts.</p>
<p>The work continues to compile as much as possible of the Tezos code
base to Javascript.
The precision surgery to remove all Unix dependencies, tweak the OCaml
signatures, write bindings and compile C libraries to <span class="caps">WASM</span> is
progressing steadily and will soon allow to reuse a huge amount of
code for the applications built around Tezos.</p>
<h1 id="crypto"><a class="toclink" href="#crypto">Crypto</a></h1>
<p>Work has started to refresh our version of the <span class="caps">HACL</span>* cryptographic
library which will provide a verified implementation of curve P-256
(tz3 addresses) on top of the existing Ed25519.</p>
<p>Moreover our new binding for the ZCash Sapling library is almost done and
we are starting to experiment with it for a future protocol proposal.</p>
<h1 id="events"><a class="toclink" href="#events">Events</a></h1>
<p>A few nomads were also spotted to some events recently.
We were happy to help <a href="http://chainaccelerator.com/">Chain
Accelerator</a> during a recent week long
training session on Tezos and we lent a hand at the <a href="https://hackathon.trustsquare.ch/">Swiss Blockchain
Hackathon in Zurich</a>.</p>
<p>Next week we are conducting a Tezos tutorial at
<a href="http://hpcs2019.cisedu.info/"><span class="caps">HPCS</span>‘19</a> and a few devs are also
preparing for the next masterclass organized by <span class="caps">TSA</span> that will take
place next week in Vietnam. Looking forward to it!</p>How to write a Tezos protocol2019-06-21T18:00:00+02:002019-06-21T18:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2019-06-21:/how-to-write-a-tezos-protocol.html<p>A Tezos node is parameterized by a software component called an
<em>economic protocol</em> (or protocol for short). Different protocol
implementations can be used to implement different types of blockchains.</p>
<p>This is the first post of a tutorial series on how to implement such a
protocol. We will see how to write, compile, register, activate and use an
extremely simple protocol. By doing so, we will also start to explore the
interface between the protocol and the node (more specifically the
<em>shell</em> component of the node). In later blog posts, we will gradually work
our way up to a more realistic protocol.</p>
<p>In what follows, we suppose you have cloned the Tezos repository and we
specifically look at revision <code>2d903a01</code> on the <code>master</code> branch. You should be
already familiar with <a href="http://tezos.gitlab.io/user/sandbox.html">running a node in sandbox mode</a>.
All paths are relative to the root of this repository. All Bash
commands are to be executed in sandbox mode.</p>
<h1 id="protocol-registration"><a class="toclink" href="#protocol-registration">Protocol Registration</a></h1>
<p>A node can contain several economics protocols (they are said to be
<em>registered</em>), but only one is <em>activated</em> at any given time.</p>
<p>We can query a node to know the registered protocols. Protocols are
identified by a b58check hash. On the <code>master</code> branch, hashes are
arbitrary values and do not depend on the actual code, but on production
branches, they are hashes of the source code of the protocol.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-admin-client<span class="w"> </span>list<span class="w"> </span>protocols
ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK
ProtoDemoNoopsDemoNoopsDemoNoopsDemoNoopsDemo6XBoYp
ProtoGenesisGenesisGenesisGenesisGenesisGenesk612im
</code></pre></div>
<p>The node in this example contains three protocols. They were statically
linked (<em>embedded</em>) to the node at compile time. <code>genesis</code> is the protocol
activated at start-up. <code>alpha</code> is the main Tezos protocol.
<code>demo_noops</code> is a simple protocol without operations (hence the name
no-ops) that we will use as our main example in this article.</p>
<p>Protocols can also be registered dynamically at run-time via an <span class="caps">RPC</span> (a.k.a.
<em>protocol injection</em>). As an example, let us inject the test protocol
<code>/src/bin_client/test/proto_test_injection</code> available as a test case in the
Tezos code base.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-admin-client<span class="w"> </span>inject<span class="w"> </span>protocol<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>src/bin_client/test/proto_test_injection
Injected<span class="w"> </span>protocol<span class="w"> </span>PshuejubNkeGc5nU2xwF7uGzCdujcZY7ZV3duFfffmG4z5SoMAM<span class="w"> </span>successfully
</code></pre></div>
<p>Under the hood, the protocol is compiled and sent to the node using
the <span class="caps">POST</span> <span class="caps">RPC</span> <code>/injection/protocol</code>.</p>
<p>We can check that the protocol was successfully injected</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-admin-client<span class="w"> </span>list<span class="w"> </span>protocols
ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK
ProtoDemoNoopsDemoNoopsDemoNoopsDemoNoopsDemo6XBoYp
ProtoGenesisGenesisGenesisGenesisGenesisGenesk612im
PshuejubNkeGc5nU2xwF7uGzCdujcZY7ZV3duFfffmG4z5SoMAM
</code></pre></div>
<p>Lastly the node can also fetch a protocol over the network, for
example before starting the test chain or activating a new amendment.
Like in the previous case, once the code is downloaded, it will be
compiled and dynamically linked.</p>
<h1 id="protocol-activation"><a class="toclink" href="#protocol-activation">Protocol Activation</a></h1>
<p>Generally, a node starts its execution with the <code>genesis</code>
protocol. <code>genesis</code> provides an operation to upgrade to a new
protocol. Interestingly, upgradability is a feature of the protocol,
not of the shell (though the shell can also force a protocol
upgrade). Protocols may or may not be upgradable. The <em>raison d’être</em>
of <code>genesis</code> is upgradability, <code>alpha</code> is upgradable by voting, while
<code>demo_noops</code> is not upgradable.</p>
<p>The client command <code>activate protocol</code> is a shorthand to craft the
activation operation offered by <code>genesis</code> to upgrade to a new protocol.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mkdir<span class="w"> </span>tmp<span class="w"> </span><span class="o">&&</span><span class="w"> </span><span class="nb">echo</span><span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="o">}</span><span class="w"> </span>><span class="w"> </span>tmp/protocol_parameters.json
$<span class="w"> </span>tezos-client<span class="w"> </span>activate<span class="w"> </span>protocol<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>ProtoDemoNoopsDemoNoopsDemoNoopsDemoNoopsDemo6XBoYp<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>with<span class="w"> </span>fitness<span class="w"> </span><span class="m">5</span><span class="w"> </span>and<span class="w"> </span>key<span class="w"> </span>activator<span class="w"> </span>and<span class="w"> </span>parameters<span class="w"> </span>tmp/protocol_parameters.json
Injected<span class="w"> </span>BMHupUhxqJqT
</code></pre></div>
<p>This command injects a so-called “activation block” to the blockchain
(the command returns the prefix of the hash of this block). This block
is the only one using the <code>genesis</code> protocol. It is a block that
contains only one operation: the operation that activates the next
protocol (in our case, <code>demo_noops</code>). The next block in the blockchain will
be the first block using the activated protocol. Let us detail the
parameters of this command:</p>
<ul>
<li><code>ProtoDemoNoopsDemoNoopsDemoNoopsDemoNoopsDemo6XBoYp</code> is the hash of
the protocol to be activated. The protocol must be registered.</li>
<li><code>activator</code> is an alias for an activation <em>secret</em> key. In this
example, the corresponding public key has been passed as a
parameter to <code>tezos-node</code> at startup (using the <code>--sandbox</code>
argument). The alias is known to the client because it is added by
default in sandbox mode.</li>
<li><code>5</code> is the fitness of the activation block (more details below). It
can be any number.</li>
<li><code>protocol_parameters.json</code> is a file that contains protocol-specific
initialization parameters. There are no parameters for <code>demo_noops</code>, so
this file contains an empty json object (i.e. <code>{ }</code>).</li>
</ul>
<p>We suggest the reader runs the following two commands to inspect the
first two blocks of the blockchain (in particular the values of the
<code>"protocol"</code>, <code>"hash"</code>, <code>"predecessor"</code>, <code>"level"</code>, and <code>"fitness"</code>
keys).</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-client<span class="w"> </span>rpc<span class="w"> </span>get<span class="w"> </span>/chains/main/blocks/head
$<span class="w"> </span>tezos-client<span class="w"> </span>rpc<span class="w"> </span>get<span class="w"> </span>/chains/main/blocks/head~1
</code></pre></div>
<h1 id="protocol-structure-and-compilation"><a class="toclink" href="#protocol-structure-and-compilation">Protocol Structure and Compilation</a></h1>
<p>Currently, the embedded protocols live in the tezos repository
besides the rest of the code. They follow the naming convention <code>proto_*</code>.
The code of <code>proto_demo_noops</code> is organized as shown below:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>ls<span class="w"> </span>-R<span class="w"> </span>src/proto_demo_noops
src/proto_demo_noops/lib_protocol:
TEZOS_PROTOCOL<span class="w"> </span>main.ml<span class="w"> </span>main.mli<span class="w"> </span>dune.inc<span class="w"> </span>tezos-protocol-demo-noops.opam<span class="w"> </span>dune
tezos-embedded-protocol-demo-noops.opam
</code></pre></div>
<p>The protocol code resides in the <code>lib_protocol</code> directory. A protocol
must define a <code>TEZOS_PROTOCOL</code> json file that contains the hash of the
protocol and the list of OCaml modules.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>cat<span class="w"> </span>src/proto_demo_noops/lib_protocol/TEZOS_PROTOCOL
<span class="o">{</span>
<span class="w"> </span><span class="s2">"hash"</span>:<span class="w"> </span><span class="s2">"ProtoDemoNoopsDemoNoopsDemoNoopsDemoNoopsDemo6XBoYp"</span>,
<span class="w"> </span><span class="s2">"modules"</span>:<span class="w"> </span><span class="o">[</span><span class="s2">"Main"</span><span class="o">]</span>
<span class="o">}</span>
</code></pre></div>
<p>Besides the <code>TEZOS_PROTOCOL</code> and <code>main.ml[i]</code> files, the other files
in <code>lib_protocol</code> are used for compiling the protocol, checking that
it respects the restrictions explained below, and for linking it with
the other components of the node.</p>
<p>Currently, protocols are compiled differently depending on whether they are
embedded or injected. Injected protocols are compiled by the Tezos compiler
embedded in <code>tezos-node</code>. Embedded protocols are compiled as <span class="caps">OPAM</span> libraries
using the standard toolchain. We leave the practical details for a future post.</p>
<h1 id="protocol-interface"><a class="toclink" href="#protocol-interface">Protocol Interface</a></h1>
<p>The economic protocol is a <em>sandboxed</em> component restricted in the
following two ways.</p>
<ul>
<li>
<p>It can only access modules defined by the <em>protocol environment</em>.</p>
</li>
<li>
<p>It must define a module <code>Main</code> which implements the interface
<code>Updater.PROTOCOL</code> from
<a href="http://tezos.gitlab.io/api/odoc/_html/tezos-protocol-environment-sigs/Tezos_protocol_environment_sigs/V0/module-type-T/Updater/index.html">src/lib_protocol_environment/sigs/v1/updater.mli</a>.
The shell interacts with the protocol through this interface.</p>
</li>
</ul>
<p>In addition, just like any other node component, the protocol can define <span class="caps">RPC</span>
services to interact with a client. We will address RPCs in a later post but
there is no difficulty here apart from getting accustomed with the Tezos <span class="caps">RPC</span> library.</p>
<h2 id="environment"><a class="toclink" href="#environment">Environment</a></h2>
<p>The environment of the protocol is a fixed set of OCaml modules
(their signatures are declared in <code>src/lib_protocol_environment/sigs/v1/</code>),
consisting in a carefully chosen subset of the OCaml standard library,
plus specialized utility modules. This form of sandboxing the protocol
ensures that the protocol code does not use unsafe functions. The
<a href="http://tezos.gitlab.io/developer/protocol_environment.html">Tezos
documentation</a>
explains in more detail the restrictions the environment imposes.</p>
<p>Any datatype used by the protocol is defined in this environment (in
particular, all modules or types mentioned below). The following is
the list of Tezos-specific modules defined by the environment that we
will be mentioning throughout this blog post: <code>Block_header</code>,
<code>Context</code>, <code>Operation</code>, and <code>Updater</code>.</p>
<h2 id="updaterprotocol"><a class="toclink" href="#updaterprotocol">Updater.<span class="caps">PROTOCOL</span></a></h2>
<p>At a very high-level, a protocol must:</p>
<ol>
<li>
<p>implement protocol-specific types, such as the type of operations
or protocol-specific block header data (in addition to the shell
generic header),</p>
</li>
<li>
<p>define under which conditions a block is a valid extension of the
current blockchain, and define an ordering on blocks to arbitrate
between concurrent extensions.</p>
</li>
</ol>
<p>For instance, in a bitcoin-like protocol, the supported operations are
transactions, and the block header data contains a <em>proof of work (PoW)</em>
stamp. A block is valid if its operations are supported by enough
funds and the PoW stamp is correct.</p>
<p>For the second point, <em>at a conceptual level</em>, the protocol defines
the function <code>apply: context -> block -> (context * fitness) option</code>
which is called whenever the node processes a block. <em>context</em>
represents the <em>protocol state</em> and <em>fitness</em> is an integer used to
compare blocks. The context is therefore protocol-specific, it may
contain, for instance, a list of accounts and their balance. It must
contain enough information to determine the validity of a new
block. The fitness defines a total ordering between blocks (and
therefore between chains). The <code>option</code> type is used here to
represent block validity: the function returns <code>None</code> when the block
is not valid, while if it is valid, it returns the block’s fitness and
the updated protocol state, obtained after applying (the operations
contained in) the block.</p>
<p>The <a href="http://tezos.gitlab.io/api/odoc/_html/tezos-protocol-environment-sigs/Tezos_protocol_environment_sigs/V0/module-type-T/Updater/index.html">signature
<code>PROTOCOL</code></a>
in module <code>Updater</code> captures these general
ideas (explained in more detail in the <a href="https://tezos.com/static/white_paper-2dc8c02267a8fb86bd67a108199441bf.pdf">Tezos white
paper</a>),
but is slightly more complex, mostly for efficiency reasons. In this
first article, we will cover only some aspects of the interface and we
will cover it more fully in later posts.</p>
<p><em>Concretely</em>, a context (represented by the type <code>Context.t</code>) is a
disk-based immutable key-value store, namely, a map from <code>string list</code>
to <code>MBytes.t</code>. Such a loosely structured datatype should accommodate
most protocols. A fitness (represented by the type <code>Fitness.t</code>) is a
list of byte arrays. A total order on blocks is obtained by comparing
their fitness first by length and then lexicographically.</p>
<p>A Tezos <em>block</em> is composed of a <em>block header</em> (of type
<code>Block_header.t</code>) and a list of operations. A block header has two
parts, a protocol-independent <em>shell header</em> (described
<a href="http://tezos.gitlab.io/whitedoc/proof_of_stake.html#shell-header">here</a>)
and a protocol-specific header, which is a byte array (with type
<code>MBytes.t</code>). Similarly, <em>operations</em> (of type <code>Operation.t</code>) have a
protocol independent shell header, and a protocol-specific header. For
instance, <code>Block_header.t</code> is defined as follows.</p>
<div class="highlight"><pre><span></span><code> <span class="k">type</span> <span class="n">t</span> <span class="o">=</span> <span class="o">{</span>
<span class="n">shell</span><span class="o">:</span> <span class="n">shell_header</span> <span class="o">;</span>
<span class="n">protocol_data</span><span class="o">:</span> <span class="nn">MBytes</span><span class="p">.</span><span class="n">t</span> <span class="o">;</span>
<span class="o">}</span>
</code></pre></div>
<p>As part of implementing the <code>PROTOCOL</code> signature, the protocol must in
particular provide concrete types for the protocol-specific block
header (<code>type block_header_data</code>) and operations (<code>type
operation_data</code>). These types are private to the protocol. The only
functions exported to the shell are encoders/decoders. This allows the
shell to serialize these types, either in binary format or in
json. Typically, the binary format is used for <span class="caps">P2P</span> communications, and
json is used for human-readable RPCs. Here is an excerpt from the
<code>PROTOCOL</code> signature where these types are declared:</p>
<div class="highlight"><pre><span></span><code> <span class="c">(** The version specific type of blocks. *)</span>
<span class="k">type</span> <span class="n">block_header_data</span>
<span class="c">(** Encoding for version specific part of block headers. *)</span>
<span class="k">val</span> <span class="n">block_header_data_encoding</span><span class="o">:</span> <span class="n">block_header_data</span> <span class="nn">Data_encoding</span><span class="p">.</span><span class="n">t</span>
<span class="c">(** A fully parsed block header. *)</span>
<span class="k">type</span> <span class="n">block_header</span> <span class="o">=</span> <span class="o">{</span>
<span class="n">shell</span><span class="o">:</span> <span class="nn">Block_header</span><span class="p">.</span><span class="n">shell_header</span> <span class="o">;</span>
<span class="n">protocol_data</span><span class="o">:</span> <span class="n">block_header_data</span> <span class="o">;</span>
<span class="o">}</span>
</code></pre></div>
<p>Note the analogy between <code>Block_header.t</code> (the shell’s view of the
block header) and <code>block_header</code> (the protocol’s view of the block header).</p>
<p>Several functions declared in the <code>PROTOCOL</code> signature realize together the
<code>apply</code> functionality: <code>begin_application</code>, <code>begin_partial_application</code>,
<code>begin_construction</code>, <code>apply_operation,</code> and <code>finalize_block</code>. A typical
<code>apply</code> is represented by a call to <code>begin_(application|construction)</code>,
followed by a sequence of calls to <code>apply_operation</code>, one for each operation
in the block, and finally a call to <code>finalize_block</code>. These functions use
values with types <code>validation_result</code> and <code>validation_state</code>. Defined by the
<code>PROTOCOL</code> signature, the type <code>validation_result</code> represents the result of a
block application, and it is a record type that contains most notably a
context and a fitness. <code>validation_state</code> is a protocol-defined datatype used
as intermediary state between applications of operations. To understand the
usage of these two types, it may be useful to consider the following
simplification of the types of the five functions mentioned:</p>
<div class="highlight"><pre><span></span><code><span class="n">begin_application</span><span class="o">:</span> <span class="nn">Context</span><span class="p">.</span><span class="n">t</span> <span class="o">-></span> <span class="n">block_header</span> <span class="o">-></span> <span class="n">validation_state</span>
<span class="n">begin_partial_application</span><span class="o">:</span> <span class="nn">Context</span><span class="p">.</span><span class="n">t</span> <span class="o">-></span> <span class="n">block_header</span> <span class="o">-></span> <span class="n">validation_state</span>
<span class="n">begin_construction</span><span class="o">:</span> <span class="nn">Context</span><span class="p">.</span><span class="n">t</span> <span class="o">-></span> <span class="o">?</span><span class="n">protocol_data</span><span class="o">:</span> <span class="n">block_header_data</span> <span class="o">-></span> <span class="n">validation_state</span>
<span class="n">apply_operation</span><span class="o">:</span> <span class="n">validation_state</span> <span class="o">-></span> <span class="n">operation</span> <span class="o">-></span> <span class="n">validation_state</span>
<span class="n">finalize_block</span><span class="o">:</span> <span class="n">validation_state</span> <span class="o">-></span> <span class="n">validation_result</span>
</code></pre></div>
<p>We briefly describe the role of these five functions:</p>
<ul>
<li>
<p><code>begin_application</code> is used when validating a block
received from the network.</p>
</li>
<li>
<p><code>begin_partial_application</code> is used when the shell receives a block
more than one level ahead of the current head (this happens, for
instance, when synchronizing a node). This function should run
quickly, as its main role is to reject invalid blocks from the chain
as early as possible.</p>
</li>
<li>
<p><code>begin_construction</code> is used by the shell when instructed to build a
block and for validating operations as they are gossiped on the
network. This two cases are distinguished by the optional
<code>protocol_data</code> argument: when only validating operations the
argument is missing, as there is no block header. In both of these
cases, the operations are not (yet) part of a block which is why the
function does not expect a shell block header.</p>
</li>
<li>
<p><code>apply_operation</code> is called after <code>begin_application</code> or
<code>begin_construction</code>, and before <code>finalize_block</code>, for each
operation in the block or in the mempool, respectively. Its role is
to validate the operation and to update the (intermediary) state accordingly.</p>
</li>
<li>
<p><code>finalize_block</code> represents the last step in a block validation
sequence. It produces the context that will be used as input for the
validation of the block’s successor candidates.</p>
</li>
</ul>
<p>Another important function in the <code>PROTOCOL</code> interface is <code>init</code>,
which is called when the protocol is activated. It takes as parameters
a context and the shell header of the last block of the previous
protocol. The context is the context corresponding to this last block,
which includes the protocol parameters given at activation time. It
returns a <code>validation_result</code>, which contains a context that is
prepared for the new protocol. Note that the new context may change
the key-value structure of the store compared with the previous
protocol. <code>init</code> is therefore responsible for making the migration of
the context from the previous protocol to the current protocol.</p>
<p>Finally, let us emphasize that the protocol is a <em>stateless</em>
component. Rather than maintaining a mutable state, it implements pure
functions that that take a state as a parameter and return a new
state. The shell is responsible to store this state between function calls.</p>
<h1 id="protocol-demo_noops"><a class="toclink" href="#protocol-demo_noops">Protocol <code>demo_noops</code></a></h1>
<p>The <code>demo_noops</code> protocol is very simple:</p>
<ul>
<li>It has no operations (hence no-ops).</li>
<li>It does not update its state, <code>context</code> is never modified.</li>
<li>The fitness of a block is the block’s level (i.e. its height in the blockchain).</li>
</ul>
<p>We now go through the types and functions which do not have a trivial
definition. First, we simply choose to have a string as the block
header. Therefore we define in <code>main.ml</code>:</p>
<div class="highlight"><pre><span></span><code> <span class="k">type</span> <span class="n">block_header_data</span> <span class="o">=</span> <span class="kt">string</span>
<span class="k">let</span> <span class="n">block_header_data_encoding</span> <span class="o">=</span>
<span class="nn">Data_encoding</span><span class="p">.</span><span class="o">(</span><span class="n">obj1</span> <span class="o">(</span><span class="n">req</span> <span class="s2">"block_header_data"</span> <span class="kt">string</span><span class="o">))</span>
</code></pre></div>
<p>For the encoding of the (protocol-specific) block header we rely on
the <code>data_encoding</code> library, explained
<a href="http://tezos.gitlab.io/developer/data_encoding.html">here</a>.</p>
<p>As there are no operations, the type of an operation header is just
<code>unit</code>. Similarly, as we do not use the other helper datatypes like
<code>block_header_metadata</code> and <code>`operation_receipt</code>, we simply set these
types to <code>unit</code>.</p>
<p>Next, we need to define a <code>validation_state</code>. We define it as record
datatype that contains a context and a fitness, because these need to
be passed to the <code>validation_result</code> returned by <code>finalize_block</code>.</p>
<div class="highlight"><pre><span></span><code> <span class="k">type</span> <span class="n">validation_state</span> <span class="o">=</span> <span class="o">{</span>
<span class="n">context</span> <span class="o">:</span> <span class="nn">Context</span><span class="p">.</span><span class="n">t</span> <span class="o">;</span>
<span class="n">fitness</span> <span class="o">:</span> <span class="nn">Fitness</span><span class="p">.</span><span class="n">t</span> <span class="o">;</span>
<span class="o">}</span>
</code></pre></div>
<p>Concerning the fitness, we assume that the protocol is instantiated
from <code>genesis</code>. Note that this may not be the case in general. <code>demo_noops</code>
could very well be instantiated from a previous protocol with a
totally different format for the fitness. The protocol should be able
to adjust to different fitness models. Here, however, we use the same
fitness model as <code>genesis</code> (and <code>alpha</code>), where the fitness has the
form <code>xx:xxxxxxxxxxxxxxxx</code>. That is, the fitness is a list of two byte
arrays, the first one (<code>xx</code>, of length 1), representing the protocol
version, and the second one encoding an <code>int64</code> number (thus of length
8). Recall that there is only one block using the <code>genesis</code>
protocol. For this block, the fitness’ first element is <code>00</code> and its
second element encodes the integer given as the fitness parameter when
activating the next protocol. In <code>demo_noops</code>, the first element is <code>01</code>
and the second element represents the level.</p>
<p>The helper functions needed to implement the fitness are as follows:</p>
<div class="highlight"><pre><span></span><code> <span class="k">let</span> <span class="n">version_number</span> <span class="o">=</span> <span class="s2">"</span><span class="se">\001</span><span class="s2">"</span>
<span class="k">let</span> <span class="n">int64_to_bytes</span> <span class="n">i</span> <span class="o">=</span>
<span class="k">let</span> <span class="n">b</span> <span class="o">=</span> <span class="nn">MBytes</span><span class="p">.</span><span class="n">create</span> <span class="mi">8</span> <span class="k">in</span>
<span class="nn">MBytes</span><span class="p">.</span><span class="n">set_int64</span> <span class="n">b</span> <span class="mi">0</span> <span class="n">i</span><span class="o">;</span>
<span class="n">b</span>
<span class="k">let</span> <span class="n">fitness_from_level</span> <span class="n">level</span> <span class="o">=</span>
<span class="o">[</span> <span class="nn">MBytes</span><span class="p">.</span><span class="n">of_string</span> <span class="n">version_number</span> <span class="o">;</span>
<span class="n">int64_to_bytes</span> <span class="n">level</span> <span class="o">]</span>
</code></pre></div>
<p>The fitness of a new block is actually set in <code>begin_construction</code>,
which has the following very simple implementation:</p>
<div class="highlight"><pre><span></span><code> <span class="k">let</span> <span class="n">begin_construction</span>
<span class="o">~</span><span class="n">chain_id</span><span class="o">:_</span>
<span class="o">~</span><span class="n">predecessor_context</span><span class="o">:</span><span class="n">context</span>
<span class="o">~</span><span class="n">predecessor_timestamp</span><span class="o">:_</span>
<span class="o">~</span><span class="n">predecessor_level</span>
<span class="o">~</span><span class="n">predecessor_fitness</span><span class="o">:_</span>
<span class="o">~</span><span class="n">predecessor</span><span class="o">:_</span>
<span class="o">~</span><span class="n">timestamp</span><span class="o">:_</span>
<span class="o">?</span><span class="n">protocol_data</span><span class="o">:_</span> <span class="bp">()</span>
<span class="o">=</span>
<span class="k">let</span> <span class="n">fitness</span> <span class="o">=</span> <span class="n">fitness_from_level</span> <span class="nn">Int64</span><span class="p">.</span><span class="o">(</span><span class="n">succ</span> <span class="o">(</span><span class="n">of_int32</span> <span class="n">predecessor_level</span><span class="o">))</span> <span class="k">in</span>
<span class="o">...</span> <span class="c">(* output a log message *)</span>
<span class="n">return</span> <span class="o">{</span> <span class="n">context</span> <span class="o">;</span> <span class="n">fitness</span> <span class="o">}</span>
</code></pre></div>
<p>The implementation of the other main functions is trivial:
<code>begin_application</code> just builds the validation state from the
predecessor context and the fitness from (the shell part of) the block
header. <code>begin_partial_application</code> behaves like <code>begin_application</code>.
<code>apply_operation</code> returns an error (however, it is never called), and
<code>finalize_block</code> builds a validation result from the validation state
by copying the context and the fitness, and setting default values for
the other fields. Most functions also record a log message which
allows one to see when these functions are called during the node’s
execution. They also show how the fitness is updated.</p>
<p>Finally, this protocol does not define any <span class="caps">RPC</span>.</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="n">rpc_services</span> <span class="o">=</span> <span class="nn">RPC_directory</span><span class="p">.</span><span class="n">empty</span>
</code></pre></div>
<h1 id="baking-a-block"><a class="toclink" href="#baking-a-block">Baking a block</a></h1>
<p>We can build a rudimentary baker simply using the RPCs provided by the node.
The <span class="caps">RPC</span> to
inject a block is <code>/injection/block</code>. However, we need to provide an
hexadecimal binary encoding of the block header. To obtain it we use
the following <span class="caps">RPC</span>:
<code>/chains/main/blocks/head/helpers/forge_block_header</code>. This <span class="caps">RPC</span>
expects as argument a json representation of the block header. The
json representation of the shell header (the protocol-independent part
of the header) can be obtained with the following <span class="caps">RPC</span>:
<code>/chains/main/blocks/head/helpers/forge_block_header</code>.</p>
<p>We will thus use the following RPCs to bake a block:</p>
<ol>
<li><code>/chains/main/blocks/head/helpers/preapply/block</code></li>
<li><code>/chains/main/blocks/head/helpers/forge_block_header</code></li>
<li><code>/injection/block</code></li>
</ol>
<p>We call the first <span class="caps">RPC</span> with the protocol hash, the protocol block header, and
the (empty) list of operations. The <span class="caps">RPC</span> service calls <code>begin_construction</code>
and <code>finalize_block</code> of the <code>demo_noops</code> protocol and returns the built (but not
injected) block in json format. Notice the json representation of the
protocol block header data <code>"block_header_data": "hello world"</code>) is the one we
defined in our implementation of <code>demo_noops</code>.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-client<span class="w"> </span>-p<span class="w"> </span>ProtoGenesisGenesisGenesisGenesisGenesisGenesk612im<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>rpc<span class="w"> </span>post<span class="w"> </span>/chains/main/blocks/head/helpers/preapply/block<span class="w"> </span>with<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s1">'{"protocol_data":</span>
<span class="s1"> {"protocol": "ProtoDemoNoopsDemoNoopsDemoNoopsDemoNoopsDemo6XBoYp",</span>
<span class="s1"> "block_header_data": "hello world"},</span>
<span class="s1"> "operations": []}'</span>
<span class="o">{</span><span class="w"> </span><span class="s2">"shell_header"</span>:
<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"level"</span>:<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="s2">"proto"</span>:<span class="w"> </span><span class="m">1</span>,
<span class="w"> </span><span class="s2">"predecessor"</span>:<span class="w"> </span><span class="s2">"BLCJ5s7SGvMzmJd7Y7jbpuNiTN5c8yz9L4Q2GtBpLaJTAHKMEoz"</span>,
<span class="w"> </span><span class="s2">"timestamp"</span>:<span class="w"> </span><span class="s2">"2019-06-21T15:35:37Z"</span>,<span class="w"> </span><span class="s2">"validation_pass"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"operations_hash"</span>:
<span class="w"> </span><span class="s2">"LLoZS2LW3rEi7KYU4ouBQtorua37aWWCtpDmv1n2x3xoKi6sVXLWp"</span>,
<span class="w"> </span><span class="s2">"fitness"</span>:<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"01"</span>,<span class="w"> </span><span class="s2">"0000000000000002"</span><span class="w"> </span><span class="o">]</span>,
<span class="w"> </span><span class="s2">"context"</span>:<span class="w"> </span><span class="s2">"CoV3MLpgMM91DbHGuqGz7uwgmMYjnh7EQSsqt1CxPqvxQpU9pczA"</span><span class="w"> </span><span class="o">}</span>,
<span class="w"> </span><span class="s2">"operations"</span>:<span class="w"> </span><span class="o">[]</span><span class="w"> </span><span class="o">}</span>
</code></pre></div>
<p><code>tezos-client</code> can use protocol-specific extensions. By default,
<code>tezos-client</code> tries to use the extension corresponding to the node’s
protocol. In our case no such extension has been given, therefore we
need to specify an extension using the <code>-p XXX</code> option, where <code>XXX</code> is
a protocol hash.</p>
<p>Now we use the second <span class="caps">RPC</span> to obtain the binary encoding of the protocol block header:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-client<span class="w"> </span>-p<span class="w"> </span>ProtoGenesisGenesisGenesisGenesisGenesisGenesk612im<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>rpc<span class="w"> </span>post<span class="w"> </span>/chains/main/blocks/head/helpers/forge_block_header<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>with<span class="w"> </span><span class="s1">'{"level": 2, "proto": 1,</span>
<span class="s1"> "predecessor": "BLCJ5s7SGvMzmJd7Y7jbpuNiTN5c8yz9L4Q2GtBpLaJTAHKMEoz",</span>
<span class="s1"> "timestamp": "2019-06-21T15:35:37Z", "validation_pass": 0,</span>
<span class="s1"> "operations_hash": "LLoZS2LW3rEi7KYU4ouBQtorua37aWWCtpDmv1n2x3xoKi6sVXLWp",</span>
<span class="s1"> "fitness": ["01", "0000000000000002"],</span>
<span class="s1"> "context": "CoV3MLpgMM91DbHGuqGz7uwgmMYjnh7EQSsqt1CxPqvxQpU9pczA",</span>
<span class="s1"> "protocol_data": "0000000b68656c6c6f20776f726c64"}'</span>
<span class="o">{</span><span class="w"> </span><span class="s2">"block"</span>:<span class="w"> </span><span class="s2">"0000000201b478f20b61340c9e8290d7b45edf057fd180891d0e0b290abc..."</span><span class="w"> </span><span class="o">}</span>
</code></pre></div>
<p>Notice the last field <code>protocol_data</code>. It must contain the binary-encoded block
header data. Remember that we specified this encoding in the protocol with</p>
<div class="highlight"><pre><span></span><code><span class="k">let</span> <span class="n">block_header_data_encoding</span> <span class="o">=</span>
<span class="nn">Data_encoding</span><span class="p">.</span><span class="o">(</span><span class="n">obj1</span> <span class="o">(</span><span class="n">req</span> <span class="s2">"block_header_data"</span> <span class="kt">string</span><span class="o">))</span>
</code></pre></div>
<p>We can compute the binary encoding on the client side, for instance using
the <code>Data_encoding</code> library, or by writing the encoder in a different language
using the public specification of the <code>Data_encoding</code> library.</p>
<p>For this example, <code>"0000000b68656c6c6f20776f726c64"</code> is the binary
encoding of <code>"hello world"</code>.</p>
<p>Finally, the last <span class="caps">RPC</span> injects the block. After the block is validated by the
protocol (the <span class="caps">RPC</span> service calls <code>begin_application</code> and <code>finalize_block</code>),
the <span class="caps">RPC</span> returns its hash.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-client<span class="w"> </span>-p<span class="w"> </span>ProtoGenesisGenesisGenesisGenesisGenesisGenesk612im<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>rpc<span class="w"> </span>post<span class="w"> </span>injection/block<span class="w"> </span>with<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="s1">'{"data": "0000000201b478f20b61340c9e8290d7b45edf057fd180891d0e0b290abc...",</span>
<span class="s1"> "operations": []}'</span>
<span class="s2">"BM6qcDPhm57sXHv1js25qcy9WESah1C3qcpKn9y8bRZzpf8s7g8"</span>
</code></pre></div>
<p>We can look at the newly created block:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-client<span class="w"> </span>-p<span class="w"> </span>ProtoGenesisGenesisGenesisGenesisGenesisGenesk612im<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>rpc<span class="w"> </span>get<span class="w"> </span>/chains/main/blocks/head/
<span class="o">{</span><span class="w"> </span><span class="s2">"protocol"</span>:<span class="w"> </span><span class="s2">"ProtoDemoNoopsDemoNoopsDemoNoopsDemoNoopsDemo6XBoYp"</span>,
<span class="w"> </span><span class="s2">"chain_id"</span>:<span class="w"> </span><span class="s2">"NetXdQprcVkpaWU"</span>,
<span class="w"> </span><span class="s2">"hash"</span>:<span class="w"> </span><span class="s2">"BMKeY5PDbm3acKDPUt7XnARkFBj5JoUDfMbBqYTYvWrGFHPt89a"</span>,
<span class="w"> </span><span class="s2">"header"</span>:
<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"level"</span>:<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="s2">"proto"</span>:<span class="w"> </span><span class="m">1</span>,
<span class="w"> </span><span class="s2">"predecessor"</span>:<span class="w"> </span><span class="s2">"BLCJ5s7SGvMzmJd7Y7jbpuNiTN5c8yz9L4Q2GtBpLaJTAHKMEoz"</span>,
<span class="w"> </span><span class="s2">"timestamp"</span>:<span class="w"> </span><span class="s2">"2019-06-21T15:35:37Z"</span>,<span class="w"> </span><span class="s2">"validation_pass"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"operations_hash"</span>:
<span class="w"> </span><span class="s2">"LLoZS2LW3rEi7KYU4ouBQtorua37aWWCtpDmv1n2x3xoKi6sVXLWp"</span>,
<span class="w"> </span><span class="s2">"fitness"</span>:<span class="w"> </span><span class="o">[</span><span class="w"> </span><span class="s2">"01"</span>,<span class="w"> </span><span class="s2">"0000000000000002"</span><span class="w"> </span><span class="o">]</span>,
<span class="w"> </span><span class="s2">"context"</span>:<span class="w"> </span><span class="s2">"CoV3MLpgMM91DbHGuqGz7uwgmMYjnh7EQSsqt1CxPqvxQpU9pczA"</span>,
<span class="w"> </span><span class="s2">"block_header_data"</span>:<span class="w"> </span><span class="s2">"hello world"</span><span class="w"> </span><span class="o">}</span>,
<span class="w"> </span><span class="s2">"metadata"</span>:
<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"protocol"</span>:<span class="w"> </span><span class="s2">"ProtoDemoNoopsDemoNoopsDemoNoopsDemoNoopsDemo6XBoYp"</span>,
<span class="w"> </span><span class="s2">"next_protocol"</span>:<span class="w"> </span><span class="s2">"ProtoDemoNoopsDemoNoopsDemoNoopsDemoNoopsDemo6XBoYp"</span>,
<span class="w"> </span><span class="s2">"test_chain_status"</span>:<span class="w"> </span><span class="o">{</span><span class="w"> </span><span class="s2">"status"</span>:<span class="w"> </span><span class="s2">"not_running"</span><span class="w"> </span><span class="o">}</span>,
<span class="w"> </span><span class="s2">"max_operations_ttl"</span>:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span><span class="s2">"max_operation_data_length"</span>:<span class="w"> </span><span class="m">0</span>,
<span class="w"> </span><span class="s2">"max_block_header_length"</span>:<span class="w"> </span><span class="m">100</span>,<span class="w"> </span><span class="s2">"max_operation_list_length"</span>:<span class="w"> </span><span class="o">[]</span><span class="w"> </span><span class="o">}</span>,
<span class="w"> </span><span class="s2">"operations"</span>:<span class="w"> </span><span class="o">[]</span><span class="w"> </span><span class="o">}</span>
</code></pre></div>
<p>For completeness, we show below the node’s trace, that is, the output
obtained when executing the following command:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>./src/bin_node/tezos-sandboxed-node.sh<span class="w"> </span><span class="m">1</span><span class="w"> </span>--connections<span class="w"> </span><span class="m">1</span>
</code></pre></div>
<p>After the node’s initialization, we see the following:</p>
<div class="highlight"><pre><span></span><code><span class="nv">Generating</span><span class="w"> </span><span class="nv">a</span><span class="w"> </span><span class="nv">new</span><span class="w"> </span><span class="nv">identity</span>...<span class="w"> </span><span class="ss">(</span><span class="nv">level</span>:<span class="w"> </span><span class="mi">0</span>.<span class="mi">00</span><span class="ss">)</span>
<span class="nv">Stored</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">new</span><span class="w"> </span><span class="nv">identity</span><span class="w"> </span><span class="ss">(</span><span class="nv">idrCKw61WKnEHc2kuqd7BZvV1V39CN</span><span class="ss">)</span><span class="w"> </span><span class="nv">into</span><span class="w"> </span><span class="s1">'/tmp/tezos-node.cJVCgcW0/identity.json'</span>.
<span class="nv">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span>:<span class="mi">34</span>:<span class="mi">22</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nv">node</span>.<span class="nv">main</span>:<span class="w"> </span><span class="nv">Starting</span><span class="w"> </span><span class="nv">the</span><span class="w"> </span><span class="nv">Tezos</span><span class="w"> </span><span class="nv">node</span>...
<span class="nv">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span>:<span class="mi">34</span>:<span class="mi">22</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nv">node</span>.<span class="nv">main</span>:<span class="w"> </span><span class="nv">No</span><span class="w"> </span><span class="nv">local</span><span class="w"> </span><span class="nv">peer</span><span class="w"> </span><span class="nv">discovery</span>.
<span class="nv">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span>:<span class="mi">34</span>:<span class="mi">22</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nv">node</span>.<span class="nv">main</span>:<span class="w"> </span><span class="nv">Peer</span><span class="err">'s global id: idrCKw61WKnEHc2kuqd7BZvV1V39CN</span>
<span class="err">Jun 21 17:34:22 - main: shell-node initialization: bootstrapping</span>
<span class="err">Jun 21 17:34:22 - main: shell-node initialization: p2p_maintain_started</span>
<span class="err">Jun 21 17:34:22 - validator.block: Worker started</span>
<span class="err">Jun 21 17:34:22 - validation_process.sequential: Initialized</span>
<span class="err">Jun 21 17:34:22 - node.validator: activate chain NetXdQprcVkpaWU</span>
<span class="err">Jun 21 17:34:22 - validator.chain_1: Worker started for NetXdQprcVkpa</span>
<span class="err">Jun 21 17:34:22 - prevalidator.NetXdQprcVkpa.ProtoGenesis_1: Worker started for NetXdQprcVkpa.ProtoGenesis</span>
<span class="err">Jun 21 17:34:22 - node.main: Starting a RPC server listening on ::ffff:127.0.0.1:18731.</span>
<span class="err">Jun 21 17:34:22 - node.main: The Tezos node is now running!</span>
</code></pre></div>
<p>After the protocol activation command, we further see the following output:</p>
<div class="highlight"><pre><span></span><code><span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">53</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">noops</span><span class="p">:</span><span class="w"> </span><span class="n">init</span><span class="p">:</span><span class="w"> </span><span class="n">fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">00</span><span class="p">::</span><span class="mi">0000000000000005</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">53</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">noops</span><span class="p">:</span><span class="w"> </span><span class="n">init</span><span class="p">:</span><span class="w"> </span><span class="n">fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">00</span><span class="p">::</span><span class="mi">0000000000000005</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">54</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">validator</span><span class="o">.</span><span class="n">block</span><span class="p">:</span><span class="w"> </span><span class="n">Block</span><span class="w"> </span><span class="n">BLCJ5s7SGvMzmJd7Y7jbpuNiTN5c8yz9L4Q2GtBpLaJTAHKMEoz</span><span class="w"> </span><span class="n">successfully</span><span class="w"> </span><span class="n">validated</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">54</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">validator</span><span class="o">.</span><span class="n">block</span><span class="p">:</span><span class="w"> </span><span class="n">Pushed</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">21</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">53</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">,</span><span class="w"> </span><span class="n">Treated</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">21</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">53</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">,</span><span class="w"> </span><span class="n">Completed</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">21</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">54</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">54</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">prevalidator</span><span class="o">.</span><span class="n">NetXdQprcVkpa</span><span class="o">.</span><span class="n">ProtoDemoNoo_1</span><span class="p">:</span><span class="w"> </span><span class="n">Worker</span><span class="w"> </span><span class="n">started</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">NetXdQprcVkpa</span><span class="o">.</span><span class="n">ProtoDemoNoo</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">54</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">noops</span><span class="p">:</span><span class="w"> </span><span class="n">begin_construction</span><span class="w"> </span><span class="p">(</span><span class="n">mempool</span><span class="p">):</span><span class="w"> </span><span class="n">pred_fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">00</span><span class="p">::</span><span class="mi">0000000000000005</span><span class="w"> </span><span class="n">constructed</span><span class="w"> </span><span class="n">fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000002</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">54</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">prevalidator</span><span class="o">.</span><span class="n">NetXdQprcVkpa</span><span class="o">.</span><span class="n">ProtoGenesis_1</span><span class="p">:</span><span class="w"> </span><span class="n">Worker</span><span class="w"> </span><span class="n">terminated</span><span class="w"> </span><span class="p">[</span><span class="n">NetXdQprcVkpa</span><span class="o">.</span><span class="n">ProtoGenesis</span><span class="p">]</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">54</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">validator</span><span class="o">.</span><span class="n">chain_1</span><span class="p">:</span><span class="w"> </span><span class="n">Update</span><span class="w"> </span><span class="n">current</span><span class="w"> </span><span class="n">head</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">BLCJ5s7SGvMzmJd7Y7jbpuNiTN5c8yz9L4Q2GtBpLaJTAHKMEoz</span><span class="w"> </span><span class="p">(</span><span class="n">fitness</span><span class="w"> </span><span class="mi">00</span><span class="p">::</span><span class="mi">0000000000000005</span><span class="p">),</span><span class="w"> </span><span class="n">same</span><span class="w"> </span><span class="n">branch</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">54</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">validator</span><span class="o">.</span><span class="n">chain_1</span><span class="p">:</span><span class="w"> </span><span class="n">Pushed</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">21</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">54</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">,</span><span class="w"> </span><span class="n">Treated</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">21</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">54</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">,</span><span class="w"> </span><span class="n">Completed</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">21</span><span class="n">T15</span><span class="p">:</span><span class="mi">34</span><span class="p">:</span><span class="mi">54</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span>
</code></pre></div>
<p>After the first <span class="caps">RPC</span>, we obtain:</p>
<div class="highlight"><pre><span></span><code><span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">35</span><span class="p">:</span><span class="mi">37</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">noops</span><span class="p">:</span><span class="w"> </span><span class="n">begin_construction</span><span class="w"> </span><span class="p">(</span><span class="n">block</span><span class="p">):</span><span class="w"> </span><span class="n">pred_fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">00</span><span class="p">::</span><span class="mi">0000000000000005</span><span class="w"> </span><span class="n">constructed</span><span class="w"> </span><span class="n">fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000002</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">35</span><span class="p">:</span><span class="mi">37</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">noops</span><span class="p">:</span><span class="w"> </span><span class="n">finalize_block</span><span class="p">:</span><span class="w"> </span><span class="n">fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000002</span>
</code></pre></div>
<p>There is no output corresponding to the second <span class="caps">RPC</span>. After
the third and last <span class="caps">RPC</span>, we see the following output:</p>
<div class="highlight"><pre><span></span><code><span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">noops</span><span class="p">:</span><span class="w"> </span><span class="n">begin_application</span><span class="p">:</span><span class="w"> </span><span class="n">pred_fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">00</span><span class="p">::</span><span class="mi">0000000000000005</span><span class="w"> </span><span class="n">block_fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000002</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">noops</span><span class="p">:</span><span class="w"> </span><span class="n">finalize_block</span><span class="p">:</span><span class="w"> </span><span class="n">fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000002</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">validator</span><span class="o">.</span><span class="n">block</span><span class="p">:</span><span class="w"> </span><span class="n">Block</span><span class="w"> </span><span class="n">BM6qcDPhm57sXHv1js25qcy9WESah1C3qcpKn9y8bRZzpf8s7g8</span><span class="w"> </span><span class="n">successfully</span><span class="w"> </span><span class="n">validated</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">validator</span><span class="o">.</span><span class="n">block</span><span class="p">:</span><span class="w"> </span><span class="n">Pushed</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">21</span><span class="n">T15</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">,</span><span class="w"> </span><span class="n">Treated</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">21</span><span class="n">T15</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">,</span><span class="w"> </span><span class="n">Completed</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">21</span><span class="n">T15</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">noops</span><span class="p">:</span><span class="w"> </span><span class="n">finalize_block</span><span class="p">:</span><span class="w"> </span><span class="n">fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000002</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">demo</span><span class="o">-</span><span class="n">noops</span><span class="p">:</span><span class="w"> </span><span class="n">begin_construction</span><span class="w"> </span><span class="p">(</span><span class="n">mempool</span><span class="p">):</span><span class="w"> </span><span class="n">pred_fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000002</span><span class="w"> </span><span class="n">constructed</span><span class="w"> </span><span class="n">fitness</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000003</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">prevalidator</span><span class="o">.</span><span class="n">NetXdQprcVkpa</span><span class="o">.</span><span class="n">ProtoDemoNoo_1</span><span class="p">:</span><span class="w"> </span><span class="n">switching</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">new</span><span class="w"> </span><span class="n">head</span><span class="w"> </span><span class="n">BM6qcDPhm57sXHv1js25qcy9WESah1C3qcpKn9y8bRZzpf8s7g8</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">prevalidator</span><span class="o">.</span><span class="n">NetXdQprcVkpa</span><span class="o">.</span><span class="n">ProtoDemoNoo_1</span><span class="p">:</span><span class="w"> </span><span class="n">Pushed</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">21</span><span class="n">T15</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">,</span><span class="w"> </span><span class="n">Treated</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">21</span><span class="n">T15</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">,</span><span class="w"> </span><span class="n">Completed</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">21</span><span class="n">T15</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">validator</span><span class="o">.</span><span class="n">chain_1</span><span class="p">:</span><span class="w"> </span><span class="n">Update</span><span class="w"> </span><span class="n">current</span><span class="w"> </span><span class="n">head</span><span class="w"> </span><span class="n">to</span><span class="w"> </span><span class="n">BM6qcDPhm57sXHv1js25qcy9WESah1C3qcpKn9y8bRZzpf8s7g8</span><span class="w"> </span><span class="p">(</span><span class="n">fitness</span><span class="w"> </span><span class="mi">01</span><span class="p">::</span><span class="mi">0000000000000002</span><span class="p">),</span><span class="w"> </span><span class="n">same</span><span class="w"> </span><span class="n">branch</span>
<span class="n">Jun</span><span class="w"> </span><span class="mi">21</span><span class="w"> </span><span class="mi">17</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">validator</span><span class="o">.</span><span class="n">chain_1</span><span class="p">:</span><span class="w"> </span><span class="n">Pushed</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">21</span><span class="n">T15</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">,</span><span class="w"> </span><span class="n">Treated</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">21</span><span class="n">T15</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">,</span><span class="w"> </span><span class="n">Completed</span><span class="p">:</span><span class="w"> </span><span class="mi">2019</span><span class="o">-</span><span class="mi">06</span><span class="o">-</span><span class="mi">21</span><span class="n">T15</span><span class="p">:</span><span class="mi">38</span><span class="p">:</span><span class="mi">35</span><span class="o">-</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span>
</code></pre></div>
<h2 id="wrapping-up-with-python"><a class="toclink" href="#wrapping-up-with-python">Wrapping up with Python</a></h2>
<p>Two python scripts summarize what we have covered in this post:
. <code>tests_python/tests/test_injection.py</code>
. <code>tests_python/examples/proto_demo_noops.py</code></p>
<p>This first one shows how to inject a new protocol in a node. The second
script launches a node in sandbox, activates the <code>demo_noops</code> protocol, and bakes a
block. See
<a href="https://gitlab.com/tezos/tezos/blob/master/tests_python/README.md">here</a> for
more details about how to run Python scripts, including installation instructions.</p>
<p>Excerpt from <code>tests_python/examples/proto_demo_noops.py</code>:</p>
<div class="highlight"><pre><span></span><code><span class="k">with</span> <span class="n">Sandbox</span><span class="p">(</span><span class="n">paths</span><span class="o">.</span><span class="n">TEZOS_HOME</span><span class="p">,</span>
<span class="n">constants</span><span class="o">.</span><span class="n">IDENTITIES</span><span class="p">,</span>
<span class="n">constants</span><span class="o">.</span><span class="n">GENESIS_PK</span><span class="p">,</span>
<span class="n">log_dir</span><span class="o">=</span><span class="s1">'tmp'</span><span class="p">)</span> <span class="k">as</span> <span class="n">sandbox</span><span class="p">:</span>
<span class="c1"># launch a sandbox node</span>
<span class="n">sandbox</span><span class="o">.</span><span class="n">add_node</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">sandbox</span><span class="o">.</span><span class="n">client</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="n">protocols</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">list_protocols</span><span class="p">()</span>
<span class="k">assert</span> <span class="n">PROTO_DEMO</span> <span class="ow">in</span> <span class="n">protocols</span>
<span class="n">parameters</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">client</span><span class="o">.</span><span class="n">activate_protocol_json</span><span class="p">(</span><span class="n">PROTO_DEMO</span><span class="p">,</span> <span class="n">parameters</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="s1">'activator'</span><span class="p">,</span>
<span class="n">fitness</span><span class="o">=</span><span class="s1">'1'</span><span class="p">)</span>
<span class="n">head</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">rpc</span><span class="p">(</span><span class="s1">'get'</span><span class="p">,</span> <span class="s1">'/chains/main/blocks/head/'</span><span class="p">,</span> <span class="n">params</span><span class="o">=</span><span class="n">PARAMS</span><span class="p">)</span>
<span class="c1"># current protocol is still genesis and level == 1</span>
<span class="k">assert</span> <span class="n">head</span><span class="p">[</span><span class="s1">'header'</span><span class="p">][</span><span class="s1">'level'</span><span class="p">]</span> <span class="o">==</span> <span class="mi">1</span>
<span class="k">assert</span> <span class="n">head</span><span class="p">[</span><span class="s1">'protocol'</span><span class="p">]</span> <span class="o">==</span> <span class="n">PROTO_GENESIS</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="c1"># bake a block for new protocol, using fours RPCs:</span>
<span class="c1"># - helpers/preapply/block builds the block</span>
<span class="c1"># - helpers/forge_block_header encodes the whole block header</span>
<span class="c1"># - /injection/block injects it</span>
<span class="n">message</span> <span class="o">=</span> <span class="s2">"hello world"</span>
<span class="n">data</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"protocol_data"</span><span class="p">:</span>
<span class="p">{</span><span class="s2">"protocol"</span><span class="p">:</span> <span class="n">PROTO_DEMO</span><span class="p">,</span> <span class="s2">"block_header_data"</span><span class="p">:</span> <span class="n">message</span><span class="p">},</span>
<span class="s2">"operations"</span><span class="p">:</span> <span class="p">[]}</span>
<span class="n">block</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">rpc</span><span class="p">(</span>
<span class="s1">'post'</span><span class="p">,</span>
<span class="s1">'/chains/main/blocks/head/helpers/preapply/block'</span><span class="p">,</span>
<span class="n">data</span><span class="o">=</span><span class="n">data</span><span class="p">,</span>
<span class="n">params</span><span class="o">=</span><span class="n">PARAMS</span><span class="p">)</span>
<span class="n">protocol_data</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'block_header_data'</span><span class="p">:</span> <span class="n">message</span><span class="p">}</span>
<span class="n">encoded</span> <span class="o">=</span> <span class="n">forge_block_header_data</span><span class="p">(</span><span class="n">protocol_data</span><span class="p">)</span>
<span class="n">shell_header</span> <span class="o">=</span> <span class="n">block</span><span class="p">[</span><span class="s1">'shell_header'</span><span class="p">]</span>
<span class="n">shell_header</span><span class="p">[</span><span class="s1">'protocol_data'</span><span class="p">]</span> <span class="o">=</span> <span class="n">encoded</span>
<span class="n">encoded</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">rpc</span><span class="p">(</span>
<span class="s1">'post'</span><span class="p">,</span>
<span class="s1">'/chains/main/blocks/head/helpers/forge_block_header'</span><span class="p">,</span>
<span class="n">data</span><span class="o">=</span><span class="n">shell_header</span><span class="p">,</span>
<span class="n">params</span><span class="o">=</span><span class="n">PARAMS</span><span class="p">)</span>
<span class="n">inject</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'data'</span><span class="p">:</span> <span class="n">encoded</span><span class="p">[</span><span class="s1">'block'</span><span class="p">],</span> <span class="s1">'operations'</span><span class="p">:</span> <span class="p">[]}</span>
<span class="n">client</span><span class="o">.</span><span class="n">rpc</span><span class="p">(</span><span class="s1">'post'</span><span class="p">,</span> <span class="s1">'/injection/block'</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">inject</span><span class="p">,</span> <span class="n">params</span><span class="o">=</span><span class="n">PARAMS</span><span class="p">)</span>
<span class="n">head</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">rpc</span><span class="p">(</span><span class="s1">'get'</span><span class="p">,</span> <span class="s1">'/chains/main/blocks/head/'</span><span class="p">,</span> <span class="n">params</span><span class="o">=</span><span class="n">PARAMS</span><span class="p">)</span>
<span class="k">assert</span> <span class="n">head</span><span class="p">[</span><span class="s1">'header'</span><span class="p">][</span><span class="s1">'level'</span><span class="p">]</span> <span class="o">==</span> <span class="mi">2</span>
</code></pre></div>
<h1 id="conclusion"><a class="toclink" href="#conclusion">Conclusion</a></h1>
<p>We saw how to write, compile, register, activate and use a simple protocol.
In the next blog post, we’ll make this protocol more realistic by adding
operations and block validation. We’ll also improve the client interface by
defining RPCs in the protocol, as well as extending the <code>tezos-client</code>
command-line interface with protocol-specific commands.</p>Emmy+: an improved consensus algorithm2019-06-10T18:00:00+02:002019-06-10T18:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2019-06-10:/emmy-an-improved-consensus-algorithm.html<p>Underlying the Tezos network is a consensus algorithm, which, for the ease of reference, we will call Emmy. Consensus ensures that participants agree on the same blockchain and on its state. The ideas behind Emmy are described in the <a href="https://www.tezos.com/static/papers/white_paper.pdf">white paper</a> and the specifics to its implementation can be found in the <a href="http://tezos.gitlab.io/whitedoc/proof_of_stake.html">documentation</a>. We recall that Emmy is a PoS based consensus with a use of endorsements to speed-up confirmation times and reduce selfish baking. This was done by defining the chain fitness as the total number of endorsements (in addition to the chain length).</p>
<p>This post introduces Emmy<sup>+</sup>, an improvement of Emmy which still guards against selfish baking but also achieves greater efficiency and stability.
Emmy<sup>+</sup> will be included in our next protocol proposal.</p>
<p>The main difference is that, in Emmy<sup>+</sup>, the number of endorsements included in a block no longer influences the fitness, but the timing of a block: the fewer endorsements a block carries the longer it takes before it can be considered valid. More precisely, a block is valid only if its timestamp has a minimal delay with respect to the previous block’s timestamp, and this minimal delay depends not only on the block’s priority but also on the number of endorsement operations included in the block.</p>
<p>As for block fitness, in Emmy<sup>+</sup>, it increases by one unit with each level. In this way, Emmy<sup>+</sup> simplifies the optimal baking strategy: the bakers used to have to choose whether to wait for more endorsements to include in their block, or to publish the block immediately, without waiting. The incentive for including more endorsements was to increase the fitness and win against unknown blocks. However, when a block was produced too late in the priority period, there was the risk that the block did not reach endorsers before the block of the next priority. In Emmy<sup>+</sup>, the baker does not need to take such a decision, because the baker cannot publish a block too early. This should lead to fewer missed blocks and endorsements and therefore faster convergence.</p>
<p>Another advantage of Emmy<sup>+</sup> is that chains that have fewer endorsements, signalling a difficulty in reaching consensus, become slower. This brings the algorithm closer to classical <span class="caps">BFT</span> consensus algorithms which have deterministic finality.</p>
<p>Finally, the changes lead to a few other simplifications, like simpler endorsement reward policy and shorter times between blocks when blocks at higher priority are missing, as we explain in more detail below.</p>
<h2 id="main-change-in-detail"><a class="toclink" href="#main-change-in-detail">Main change in detail</a></h2>
<p>In Emmy, the minimal delay for a block with respect to the block at the previous level is given by the following function:</p>
<div class="highlight"><pre><span></span><code> emmy_delay(p) = 60 + 75 * p
</code></pre></div>
<p>where <code>p</code> is the block’s priority and the values <code>60</code> and <code>75</code> are given by the <code>time_between_blocks</code> constant, set to <code>[60; 75]</code>. (In the code base, the definition of what we call <code>emmy_delay</code> is implemented by the function <code>Baking.minimum_time</code>; see <code>src/proto_alpha/lib_protocol/baking.ml</code> in the <code>master</code> branch).</p>
<p>In Emmy<sup>+</sup>, the new delay function depends not only on priority but also on the block’s <em>endorsing power</em> <code>e</code>, that is, the number of slots corresponding to the endorsements included in the block. This new delay function is defined as follows.</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="nv">emmy_plus_delay</span><span class="ss">(</span><span class="nv">p</span>,<span class="w"> </span><span class="nv">e</span><span class="ss">)</span><span class="w"> </span><span class="o">=</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nv">e</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="nv">initial_endorsers</span><span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nv">emmy_delay</span><span class="ss">(</span><span class="nv">p</span><span class="ss">)</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="nv">emmy_delay</span><span class="ss">(</span><span class="nv">p</span><span class="ss">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nv">delay_per_missing_endorsement</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="ss">(</span><span class="nv">initial_endorsers</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nv">e</span><span class="ss">)</span>
</code></pre></div>
<p>where <code>delay_per_missing_endorsement</code> and <code>initial_endorsers</code> are two new constants.</p>
<p>The value of <code>initial_endorsers</code> (<code>init_endo</code>) is set to <code>18</code>. Recall that the maximum endorsing power of a block is given by the <code>endorsers_per_block</code> constant, set to <code>32</code>. The rationale behind choosing the value <code>18</code> will be explained in a technical blog post and is intended to minimize the ability of a malicious baker to steal slots.</p>
<p>As for <code>delay_per_missing_endorsement</code>, it denotes a time period (<code>Period.t</code>) and it is set to <code>5</code> seconds. Therefore, with each missing endorsement with respect to the initial required amount, 5 additional seconds need to pass before the baker can send its block.</p>
<p>In addition, the value of <code>time_between_blocks</code> is changed from <code>[60; 75]</code> to <code>[60; 40]</code>. Thus the time between blocks when the block at priority 0 is missing is shortened.</p>
<p>The new delay function can be visualized on the following graph:
<img src="images/delay_fct_18_40_5.png" style="display:block;margin:auto;width:50%;" alt="delay function for Emmy+"></p>
<p>The graph depicts 5 functions: <code>t = emmy_plus_delay(p, e)</code> for 5 values of <code>p</code>. It can be read as follows: a block at priority <code>p</code> and with endorsing power <code>e</code> needs to have a timestamp at least <code>t</code> seconds bigger than the timestamp of the block at the previous level. For instance, a block at priority <code>0</code> with endorsement power <code>12</code>, needs to have at least <code>60 + 5 * max(0, 18 - 12) = 90</code> seconds delay, while a block at priority <code>2</code> with endorsement power <code>24</code> needs to have at least <code>60 + 2 * 40 + 5 * max(0, 18 - 24) = 140</code> seconds delay.</p>
<p>Finally, we emphasize that, besides minimizing the timing between blocks, these constants are also chosen such that a baker cannot steal the baking slots of other bakers by withholding its endorsements (unless it controls significantly more than one third of the total stake). We will publish a more technical blog post with details of our analysis, which will also show how the constants were chosen. The constants might actually slightly change as we refine our analysis.</p>
<h2 id="changes-to-the-reward-system"><a class="toclink" href="#changes-to-the-reward-system">Changes to the reward system</a></h2>
<p>The rewards for blocks and endorsements are now dependent on blocks’ priority: for priority 0, the same rewards as in Emmy apply, however, for larger priorities, the block rewards become smaller and endorsement rewards are computed differently.</p>
<p>Concretely, the baking reward becomes <code>block_reward / (p+1)</code> and the reward per endorsement becomes <code>endorsement_reward / (p+1)</code> where <code>p</code> is the baked block priority or the priority of the block which contains the endorsement, respectively. Note that in Emmy, for endorsements, <code>p</code> was the priority of the endorsed block, not of the block which includes the endorsement. The constants <code>block_reward</code> and <code>endorsement_reward</code> do not change, they are still set to <code>16</code> and <code>2</code> tezzies. Security deposits also do not change.</p>
<p>This change makes selfish baking attempts much less profitable: a baker with a slot at priority 1 who might attempt to steal the block at priority 0 by withholding his endorsements would be better off having his endorsements included in the block of priority 0.</p>
<h2 id="other-changes"><a class="toclink" href="#other-changes">Other changes</a></h2>
<h3 id="rpc-additions"><a class="toclink" href="#rpc-additions"><span class="caps">RPC</span> additions</a></h3>
<p>There are three additional <span class="caps">RPC</span> services:</p>
<ul>
<li>
<p><code>POST chains/<chain>/blocks/<block>/endorsing_power</code> returns the endorsement power (i.e number of slots) of a given endorsement operation.</p>
</li>
<li>
<p><code>GET chains/<chain>/blocks/<block>/required_endorsements?[block_delay=<int64>]</code> takes a block delay which represents the delay of a block’s timestamp with respect to the minimum time to bake at the block’s priority (as returned by <code>Baking.minimum_time</code> or <code>emmy_delay</code> above). It returns the minimum number of endorsements that the block has to contain to be valid.</p>
</li>
<li>
<p><code>GET chains/<chain>/blocks/<block>/minimal_valid_time?[priority=<int>]&[endorsing_power=<int>]</code> takes two arguments, a priority and an endorsing power, and it returns the minimum time at which the next block can be baked. This is the dual of the previous service. It corresponds to the <code>emmy_plus_delay</code> function given above.</p>
</li>
</ul>
<h3 id="improvements-to-the-baker"><a class="toclink" href="#improvements-to-the-baker">Improvements to the baker</a></h3>
<p>The <code>--await_endorsements</code> argument to the baker was removed, as it becomes unnecessary. Indeed, the baker is forced to wait for a minimal amount of endorsements, it does not have a choice any longer.</p>
<h3 id="new-field-in-endorsement-operations"><a class="toclink" href="#new-field-in-endorsement-operations">New field in endorsement operations</a></h3>
<p>Now an endorsement operation also includes a <code>slot</code> field, whose value is the first slot of the endorser. This is done in order to determine the endorser more quickly. Previously, the endorser was determined by iterating through all slot owners and checking which public key matched the endorsement’s signature; which was not an efficient method.</p>
<h2 id="emmy-in-the-next-protocol-proposal"><a class="toclink" href="#emmy-in-the-next-protocol-proposal">Emmy<sup>+</sup> in the next protocol proposal</a></h2>
<p>Emmy<sup>+</sup> will be part of the next proposal of a protocol amendment. The mentioned changes appear in this <a href="https://gitlab.com/nomadic-labs/tezos/-/merge_requests/58">merge request</a>.</p>Release of Mainnet May2019-05-23T18:00:00+02:002019-05-23T18:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2019-05-23:/release-of-mainnet-may.html<p>Today we are proud to announce a new release of the Tezos node and client.</p>
<p>The main features present in this release are:</p>
<ul>
<li>node support for snapshots (boot and synchronize a node in minutes),</li>
<li>node support for new history modes (not everyone needs to be an archive),</li>
<li>client integration for a multi-signature contract.</li>
</ul>
<p>Remember that besides major features there are always a myriad of
smaller improvements, have a look at the
<a href="http://tezos.gitlab.io/releases/may-2019.html#changelog">changelog</a>.</p>
<p>We encourage bakers to switch to history mode “full” using a snapshot
before the activation of Athens to speedup the validation of
the transition block.
Detailed instructions are available on the <a href="http://tezos.gitlab.io/releases/may-2019.html#storage-migration">release
page</a>.</p>
<h1 id="history-modes"><a class="toclink" href="#history-modes">History modes</a></h1>
<p><a href="https://gitlab.com/tezos/tezos/-/merge_requests/1067">Gitlab <span class="caps">MR</span></a></p>
<p>History mode, together with snapshots, is a feature that has been in
the works for quite some time.
For a full dive have a look at the original <a href="https://research-development.nomadic-labs.com/introducing-snapshots-and-history-modes-for-the-tezos-node.html">in-depth
post</a>.
In a nutshell, the Tezos node now supports two additional history
modes other than <code>archive</code>, allowing for a drastically reduced disk usage.</p>
<p>The <code>archive</code> mode keeps the whole history of the chain, that is all
the blocks and all the contexts resulting from their application.
For example it is possible to check that the block at level n had a
transfer from Alice to Bob and query that the balance of both accounts
in the context were updated.
This mode is particularly useful for a block explorer or an
application that needs to query the past.</p>
<p>History mode <code>full</code> still has the whole history of the chain but only
keeps the blocks.
This means that it is possible to replay the whole history from
genesis and verify the integrity of the chain but it’s not
possible to inspect a context far in the past.
In our example we would be able to find the transaction from Alice to
Bob at level n, but not query the effect on their balances.
Note however that, if needed, it is possible to reconstruct an archive
node from a full node by replaying the application of each block and
obtaining each intermediate context.
This mode is the new default for all newly deployed nodes.</p>
<p>History mode <code>rolling</code> pushes the concept further and permits to
delete altogether all the data from a given cutoff point.
In this case it’s not possible to access the entirety of the chain
from genesis but the disk footprint is reduced even further.</p>
<p>At this time we <strong>discourage baking from a rolling node</strong> as not all
necessary information might be available.</p>
<p><strong>Baking from a full node is supported and recommended.</strong> Indeed, not only
does the baker performance benefit from the reduced disk usage, but it
also has no need for the archives. If a baking service needs the
archives, they should consider deploying both a full node for baking,
and an archive node for other purposes.</p>
<p>For more details refer to the
<a href="http://tezos.gitlab.io/user/history_modes.html">documentation</a>.</p>
<h1 id="snapshots"><a class="toclink" href="#snapshots">Snapshots</a></h1>
<p><a href="https://gitlab.com/tezos/tezos/-/merge_requests/1069">Gitlab <span class="caps">MR</span></a></p>
<p>Snapshots are a compact and portable way to store the status of a Tezos chain.
They can be easily transmitted and used to setup new nodes without the
need to re-synchronize the chain as showcased in a recent
<a href="https://medium.com/tezos/https-medium-com-tezos-spinning-up-a-tezos-node-in-under-a-few-minutes-3b3328e173f3">blogpost</a>
from Rajath Alex of <span class="caps">TQ</span>.</p>
<p>When importing a snapshot from a third party, it is particularly
important to verify that the block hash of the snapshot is correct and
consistent with respect to several independent sources (block
explorers, wallets, …).
An attacker could provide a snapshot representing an alternative chain
history to fool your node.</p>
<p>It is good practice to regularly take a snapshot of a node, for
example with a cron job every night or week.
The files produce are relatively small, especially when compressed. As
of today a full snapshot takes 1.<span class="caps">8GB</span> (1.1 gzipped) while a rolling
snapshot takes <span class="caps">417MB</span> (137 gzipped).</p>
<p>For more details refer to the
<a href="http://tezos.gitlab.io/user/snapshots.html">documentation</a>.</p>
<h1 id="multisig"><a class="toclink" href="#multisig">Multisig</a></h1>
<p><a href="https://gitlab.com/tezos/tezos/-/merge_requests/1010">Gitlab <span class="caps">MR</span></a></p>
<p>Tezos client now integrates a few commands to originate and interact
with the <a href="https://github.com/murbard/smart-contracts/blob/master/multisig/michelson/multisig.tz">multi-signature smart
contract</a>
written in Michelson.
The client supports originating the contract with the specified keys
and the required threshold to unlock the funds.
It is also possible to change existing keys, remove them or change the
threshold.
Being only a friendlier interface to a standard smart contract, it is
still possible to interact with the multisig by performing a normal
transaction from your favorite client or wallet.</p>
<p>More importantly, the contract has been formally proven to comply with
the functional specification of a multisig using our Coq framework
<a href="https://gitlab.com/nomadic-labs/mi-cho-coq/">Mi-Cho-Coq</a>.
While this verification does not exclude the possibility of bugs in the
specification or the Michelson interpreter, it gives a very high level of
confidence in the correctness of this smart-contract.
More details on the multisig and its verification will be available in
future posts.</p>
<p>Originating a M-of-N multisig in Tezos is now as easy as:</p>
<div class="highlight"><pre><span></span><code><span class="nv">tezos</span><span class="o">-</span><span class="nv">client</span><span class="w"> </span><span class="nv">deploy</span><span class="w"> </span><span class="nv">multisig</span><span class="w"> </span><span class="o"><</span><span class="nv">contract</span><span class="o">-</span><span class="nv">alias</span><span class="o">></span><span class="w"> </span>\
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="o"><</span><span class="nv">manager</span><span class="o">-</span><span class="nv">account</span><span class="o">></span><span class="w"> </span>\
<span class="w"> </span><span class="nv">transferring</span><span class="w"> </span><span class="o"><</span><span class="nv">amount</span><span class="o">></span><span class="w"> </span><span class="nv">from</span><span class="w"> </span><span class="o"><</span><span class="nv">source</span><span class="o">-</span><span class="nv">account</span><span class="o">></span><span class="w"> </span>\
<span class="w"> </span><span class="nv">with</span><span class="w"> </span><span class="nv">threshold</span><span class="w"> </span><span class="o"><</span><span class="nv">M</span><span class="o">></span><span class="w"> </span>\
<span class="w"> </span><span class="nv">on</span><span class="w"> </span><span class="nv">public</span><span class="w"> </span><span class="nv">keys</span><span class="w"> </span><span class="o"><</span><span class="nv">signer</span><span class="o">-</span><span class="mi">1</span><span class="w"> </span>...<span class="w"> </span><span class="nv">signer</span><span class="o">-</span><span class="nv">N</span><span class="o">></span><span class="w"> </span>\
<span class="w"> </span><span class="o">--</span><span class="nv">burn</span><span class="o">-</span><span class="nv">cap</span><span class="w"> </span><span class="mi">100</span>
</code></pre></div>Athens on the testchain2019-04-05T19:00:00+02:002019-04-05T19:00:00+02:00Nomadic Labstag:research-development.nomadic-labs.com,2019-04-05:/athens-on-the-testchain.html<p>The quorum for the second voting phase has been reached, with a supermajority of <code>Yay</code>s!
This means that the voting procedure can proceed to its third stage:
the test chain for Athens is going to be launched soon, approximately Friday April 12th.
To accompany that, we just released a new version of the Tezos node and its baking daemons.
This release includes many improvements, most notably a new <code>--enable-testchain</code> option.
Users who want to participate in the test chain should
upgrade to this release and run a node with that option.
As this is the first time this procedure is run in Mainnet,
we recommend that node administrators enable this option on a secondary deployment only.</p>
<h2 id="using-the-test-chain"><a class="toclink" href="#using-the-test-chain">Using the test chain</a></h2>
<p>The main interest of running a testchain that forks from a mainchain,
as opposed to running a separate test network such as Alphanet or
Zeronet, is twofold.</p>
<p>First, we can see the effect of the context migration on a real
context which is much more complex than any test network context.</p>
<p>Second, we can actually perform operations using real accounts, which
allows testing remote signers, hardware wallets etc.</p>
<p>For this, the <code>tezos-client</code> can now take a global parameter <code>--chain
test</code>. All operations made with this parameter will only be valid in
the test chain. At the protocol level, each operation is signed with a
unique identifier for each network so it is not possible to replay
operations from the test chain on the main chain.</p>
<p>Similarly, developers can use the <code>/chains/test</code> <span class="caps">API</span> prefix, in lieu
of the usual <code>/chains/main</code>.</p>
<p>Both the option and the <span class="caps">API</span> prefix are only valid when the test chain
is active.</p>
<h2 id="running-a-lighter-test-chain-infrastructure-from-a-snapshot"><a class="toclink" href="#running-a-lighter-test-chain-infrastructure-from-a-snapshot">Running a lighter test chain infrastructure from a snapshot</a></h2>
<p>We recently released a version of the Tezos node with <a href="https://research-development.nomadic-labs.com/introducing-snapshots-and-history-modes-for-the-tezos-node.html">snapshots and
history modes</a>, available for beta
testing in the <code>mainnet-snapshots</code> Git branch/docker tag.</p>
<p>To help deploy a secondary infrastructure, this node release includes
a new command <code>snapshot export</code>. This way, node administrators
can extract a snapshot from an existing deployment, in order to spawn
a secondary one for running the test chain with a much lower footprint.
It is also a good opportunity to get familiar with snapshots, and help
us beta-test that feature.</p>
<h2 id="observing-athens-running-in-the-test-chain"><a class="toclink" href="#observing-athens-running-in-the-test-chain">Observing Athens running in the test chain</a></h2>
<p>The are several changes proposed in the variant of protocol
<code>athens_004</code> that is going to be tested, or more precisely
<code>Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd</code>.
The main changes concern 3 constants, namely the size of a roll, the
gas limit per block and per operation.
These changes can be found in the context once the test chain is
started using the rpc</p>
<div class="highlight"><pre><span></span><code><span class="n">tezos</span><span class="o">-</span><span class="n">client</span><span class="w"> </span><span class="n">rpc</span><span class="w"> </span><span class="n">get</span><span class="w"> </span><span class="o">/</span><span class="n">chains</span><span class="o">/</span><span class="n">test</span><span class="o">/</span><span class="n">blocks</span><span class="o">/</span><span class="n">head</span><span class="o">/</span><span class="n">context</span><span class="o">/</span><span class="n">constants</span>
</code></pre></div>
<p>and checking that the value of <code>tokens_per_roll</code> is indeed
<code>8000000000</code>μꜩ (or 8000ꜩ), <code>hard_gas_limit_per_operation</code>
and <code>hard_gas_limit_per_block</code> are <code>8000000</code>.</p>
<p>During the context migration there will also be a first example of invoicing, a
way for developers to fund their work on new protocols.
For the first vote Nomadic is asking for 100ꜩ, enough for a round of
drinks for the devs. This can easily be checked using tezos-client.</p>
<div class="highlight"><pre><span></span><code>tezos-client --chain test get balance tz1iSQEcaGpUn6EW5uAy3XhPiNg7BHMnRSXi
</code></pre></div>
<h2 id="baking-on-the-test-chain"><a class="toclink" href="#baking-on-the-test-chain">Baking on the test chain</a></h2>
<p>Bakers who want to bake on the test chain will also have to start the
daemons for proto-004 with the option <code>--chain test</code> on this node.</p>
<p>Several precautions are in place to avoid double baking, they have
been all updated to support the test chain, including the Ledger apps
from Obsidian Systems. Yet, as always, bakers should be extra careful
with double baking.
The <a href="http://tezos.gitlab.io/releases/april-2019.html">documentation</a>
includes specific instructions and setup recommendations.</p>Meanwhile at Nomadic Labs #22019-03-21T09:00:00+01:002019-03-21T09:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2019-03-21:/meanwhile-at-nomadic-labs-2.html<p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>The last few weeks have been pretty intense at Nomadic with the
preparation of the first community voted upgrade! The team has
been working hard on some special future improvements to Tezos…</p>
<h2 id="improvements-to-the-consensus-layer"><a class="toclink" href="#improvements-to-the-consensus-layer">Improvements to the consensus layer</a></h2>
<p>Tezos has the unique ability to amend itself which allows us
to propose state of the art research to the network at any time.
This is not a far fetched ideal and will happen sooner rather
than later in a series of new efforts spawned at Nomadic
dedicated to improving the underlying technology powering the network.</p>
<p>Two of the most prominent paths we are currently exploring relate
to consensus.</p>
<p>The first is a refinement of the current protocol, Emmy, by
changing the way endorsements are included in a block. This
improvement reduces block and endorsement misses, provides
expeditious confirmation times and better metrics relating to
block confirmation. It is our hope that this particular
update will make it into our proposal for the next voting cycle.</p>
<p>The second and more hardcore route is a complete rework of the
consensus layer in order to achieve faster finality in a more
decisive fashion. We are currently studying the adaptation of
classical <span class="caps">BFT</span> consensus algorithms to the liquid proof-of-stake
system inherent in Tezos.</p>
<p>The Nomadic team is working on a prototype of this new protocol based
on Tendermint in collaboration with the French research center (<span class="caps">CEA</span>),
Cryptium Labs and Arthur Breitman.</p>
<h2 id="introducing-mi-cho-coq"><a class="toclink" href="#introducing-mi-cho-coq">Introducing Mi-Cho-Coq</a></h2>
<p>A large subset of the Nomadic Labs team have a strong background in
programming language theory and formal verification. Until now most
of our efforts have been focused on maturing the node and its
infrastructure but this month we have kickstarted a team dedicated
solely to formal verification.</p>
<p>Their first product is
<a href="https://gitlab.com/nomadic-labs/mi-cho-coq/">Mi-Cho-Coq</a>: a framework
for the Coq proof assistant to specify Michelson smart contracts and
prove properties about them.</p>
<p>Another goal of this effort is to build a repository of simple and
idiomatic Michelson contracts for common use cases, with associated
proofs. The first one we studied is a <a href="https://github.com/murbard/smart-contracts/blob/master/multisig/michelson/multisig.tz">multi-signature Michelson
contract</a>
written by Arthur Breitman. An upcoming post will provide an in-depth
review of its proof.</p>
<h2 id="upcoming-in-depth-articles"><a class="toclink" href="#upcoming-in-depth-articles">Upcoming in-depth articles</a></h2>
<p>Topics for the next in-depth technical articles include:</p>
<ul>
<li>Proving a Multisig contract using Mi-Cho-Coq</li>
<li>How to Implement Your Own Protocol</li>
<li>Benchmarking Michelson to Get Accurate Gas Accounting</li>
</ul>Athens: Second Voting Period2019-03-20T19:00:00+01:002019-03-20T19:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2019-03-20:/athens-second-voting-period.html<p>As we approach the completion of the first active voting period, the
<code>proposal period</code>, we would like to describe what’s to come during the
second voting period. The first voting period saw the <a href="https://research-development.nomadic-labs.com/athens-proposals-injected.html">Athens
protocols injection</a> and bakers
upvoted Athens A and/or Athens B, with Athens A, the most popular
proposal, promoted for the second voting phase: the <code>test voting
period</code>. In this new period, participants are asked to vote on whether
to advance Athens A to the next stage: the testing period. Similarly
as the first period, all participants will have 8 cycles, or just over
3 weeks, to submit their voting operation on the Tezos blockchain.</p>
<p>In this post, we will provide the instructions for participants to
vote for the promotion of the current proposal to the testing period.</p>
<p>We also wish to remind Tezos participants that real-life amendment
testing is currently happening on the Zeronet test network. We still
encourage volunteers to actively participate using the instructions
described in <a href="https://research-development.nomadic-labs.com/athens-proposals-injected.html">Nomadic’s previous
blogpost</a>.</p>
<p><img src="images/yay.png" style="display:block;margin:auto;width:50%;" alt="Yay!"></p>
<h1 id="how-to-vote-in-favor-or-against-the-first-periods-winning-proposal"><a class="toclink" href="#how-to-vote-in-favor-or-against-the-first-periods-winning-proposal">How to vote in favor or against the first period’s winning proposal</a></h1>
<p>In the first period, participants could upvote Athens A, Athens B or
even submit an entirely new protocol proposal themselves. Athens A
received the greater number of votes and has been promoted to the next
period. This second period is called the <code>test voting period</code>. Its
role is to determine if the protocol is considered worthy enough to
enter the <code>testing period</code> which will be used to test the protocol in
a side-chain.</p>
<p>In the following example, we show the different commands you may use
to vote. As a reminder, Athens A hash is
<code>Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd</code> and its content
is available <a href="files/Athens_proposal_A.tar">in this tar
archive</a>.</p>
<p>Very similarly to the first period, you can create and sign voting
operation using the tezos-client:</p>
<ul>
<li>
<p>If you wish to vote <em>for</em> the promotion of this protocol, you can enter :
<code>$ tezos-client submit ballot for <delegate> Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd yay</code></p>
</li>
<li>
<p>If you wish to vote <em>against</em> the promotion of this protocol, you can enter :
<code>$ tezos-client submit ballot for <delegate> Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd nay</code></p>
</li>
<li>
<p>If you don’t wish to voice an opinion but still want to participate towards the quorum, you can enter :
<code>$ tezos-client submit ballot for <delegate> Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd pass</code></p>
</li>
</ul>
<p>You can also use the client to display the current period’s progress:</p>
<div class="highlight"><pre><span></span><code><span class="gp">$ </span>tezos-client<span class="w"> </span>show<span class="w"> </span>voting<span class="w"> </span>period<span class="sb">`</span>
<span class="go">Current period: "testing_vote"</span>
<span class="go">Blocks remaining until end of period: 1234</span>
<span class="go">Current proposal: Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd</span>
<span class="go">Ballots: { "yay": 1200, "nay": 0, "pass": 0 }</span>
<span class="go">Current participation 60.00%, necessary quorum 80.00%</span>
<span class="go">Current in favor 1200, needed supermajority 960</span>
</code></pre></div>
<h1 id="what-will-come-next"><a class="toclink" href="#what-will-come-next">What will come next</a></h1>
<p>At the end of this testing vote period in about three weeks, the
current proposal will be promoted to testing if the participation rate
(total of Yay, Nay, and Pass as a percentage of total rolls) is
greater than 80% <em>and</em> a supermajority of 80% <code>yay</code> is met. <code>pass</code>
votes also count towards the participation rate, but not in
determining the supermajority, which is based on the ratio of ‘yay’ to
‘nay’ votes. If these conditions are not met, then the amendment
process will be revert to the first period. In-depth details may be
found in the <a href="http://tezos.gitlab.io/whitedoc/voting.html">Tezos
documentation</a>.</p>
<p>Following this second period, there are still two voting phases left
provided that this period is successful:</p>
<ul>
<li>
<p>The testing period which will spawn a testchain (alongside the main
one) running the selected protocol and allowing participants to verify
the correctness of the new protocol;</p>
</li>
<li>
<p>The promotion period which will again require participants to vote
in order to finally adopt the new protocol.</p>
</li>
</ul>Athens: Proposals Injected!2019-02-28T17:00:00+01:002019-02-28T17:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2019-02-28:/athens-proposals-injected.html<p>Summary:</p>
<p>Today marks an important milestone for Tezos. We just triggered the
beginning of the first on-chain vote for self amendment. This process
could end in the successful migration from current protocol <code>alpha</code>
to <code>Athens</code> in about three months, if the participants decide so.</p>
<p>As advertised in the last <em>meanwhile …</em></p><p>Summary:</p>
<p>Today marks an important milestone for Tezos. We just triggered the
beginning of the first on-chain vote for self amendment. This process
could end in the successful migration from current protocol <code>alpha</code>
to <code>Athens</code> in about three months, if the participants decide so.</p>
<p>As advertised in the last <em>meanwhile at Nomadic</em> and as detailed in
our previous post, we injected the hashes of two proposals. Both
include the same enhancements except for one: the decreased amount in
the number of tokens needed to be your own baker (the roll size).</p>
<p>This post includes instructions to download and observe the code, to
recompute the hash of each proposal by yourself, and to examine the
differences between the two, and with the current protocol.</p>
<p>It then presents the method we recommend to bakers if they wish to
upvote either of the two proposals, and a quick reminder of what they
will have to do next to confirm their choice in the next voting phases.</p>
<p>Finally, it gives instructions for people who want to volunteer in a live
test on the Zeronet test network.</p>
<p><img src="images/athenian_token_1.png" style="display:block;margin:auto;width:50%;" alt="an Athenian voting token"></p>
<h1 id="getting-the-source-code-of-both-proposals"><a class="toclink" href="#getting-the-source-code-of-both-proposals">Getting the source code of both proposals</a></h1>
<p>We just published the source code of both proposals for Athens: <code>A</code>,
the one that includes the change in roll size from 10,000 to 8,000
tez and <code>B</code> the one without the roll size change. This code is exactly
what we documented in our <a href="https://research-development.nomadic-labs.com/athens-our-proposals-for-the-first-voted-amendment.html">previous blog
post</a>.
<code>B</code> simply does not include the change to roll size.</p>
<p>The source code of proposal <code>A</code> is available <a href="files/Athens_proposal_A.tar">in this tar
archive</a>.</p>
<p>To compute the hash, you can use the <code>tezos-protocol-compiler</code> as
follows. The result should be <code>Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd</code>.</p>
<div class="highlight"><pre><span></span><code>mkdir proposal_A
curl https://research-development.nomadic-labs.com/files/Athens_proposal_A.tar | tar x -C proposal_A
tezos-protocol-compiler -hash-only proposal_A
</code></pre></div>
<p>Similarly a tar archive for proposal <code>B</code> is located
<a href="files/Athens_proposal_B.tar">here</a>. Its
hash should be <code>Psd1ynUBhMZAeajwcZJAeq5NrxorM6UCU4GJqxZ7Bx2e9vUWB6z</code>.</p>
<p>You can observe the differences between the two by using the standard
Unix tool <code>diff proposal_A proposal_B</code>. It should only mention the
roll size change.</p>
<p>If you want to observe the changes with the current protocol, you can
use <code>diff</code> against folder <code>proto_003_PsddFKi3</code> of the mainnet source
code. All the changes should be the ones detailed in our previous blog
post, except for the removal of the single use migration code from the
previous upgrade (from <code>002_PsYLVpVv</code> to <code>003_PsddFKi3</code>).</p>
<h1 id="how-to-upvote-either-of-the-proposals"><a class="toclink" href="#how-to-upvote-either-of-the-proposals">How to upvote either of the proposals</a></h1>
<p>The easiest way to create and sign a voting operation today is to use
the command line Tezos client, as follows, and as detailed in the
<a href="http://tezos.gitlab.io/whitedoc/voting.html#client-commands">online
documentation</a>.</p>
<p>If you choose to upvote proposal <code>A</code>:</p>
<p><code>tezos-client submit proposals for <delegate> Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd</code></p>
<p>If you choose to upvote proposal <code>B</code>:</p>
<p><code>tezos-client submit proposals for <delegate> Psd1ynUBhMZAeajwcZJAeq5NrxorM6UCU4GJqxZ7Bx2e9vUWB6z</code></p>
<p>It is worth noting that these voting commands have been merged in the
mainnet distribution only recently, and bakers who want to use them
have to update their <code>tezos-client</code> to the most recent version. This
update also includes patches from Obsidian Systems to vote with a
Ledger device. No update is needed for the node or
protocol: if you have a running and synchronised node, you do not need
to restart it.</p>
<h1 id="what-will-come-next"><a class="toclink" href="#what-will-come-next">What will come next</a></h1>
<p>The current voting phase started on the 25th of February. At the end
of this period, which should be in about three weeks, one proposal
will have received a higher number of upvotes, and will be put to a
direct vote.</p>
<p>At this point, bakers will have about three weeks to vote <code>yay</code>, <code>nay</code>
or <code>pass</code> on that proposal. It is very important that they do
participate, as the quorum must be reached for the amendment process
to continue.</p>
<p>During these three weeks, we will publish an optional update of the
node that will serve two purposes.</p>
<ul>
<li>First, it will embed the most upvoted candidate and the associated
client and baking daemons, so that bakers can prepare themselves in
advance to be ready to bake when the migration comes.</li>
<li>Second, this update will include fixes to the node that are not
strictly necessary for the network to operate and retrocompatible,
but required for the third phase, the testing period, to succeed.</li>
</ul>
<p>Let’s recall that during this third phase, the network spawns a test
chain: a fork of the main chain with an expiration date (of 48 hours) and thus no
value. Bakers are encouraged to bake on this test chain to try the new
protocol in real conditions, with their actual baking keys. The
current code for the node does not spawn this test chain. This has no
incidence of the main chain, but deprives us of a useful testing mechanism.</p>
<p>Bakers who want to participate in the test chain will have to upgrade
their nodes at this point. Our plan is to include the final baker for
the main chain, and hopefully additional non vote related features
such as snapshots, so as to limit the number of upgrades for bakers.</p>
<p>Finally, if everything goes well, bakers will be ready and vote <code>yay</code>
in the last voting phase, and the network will self amend.</p>
<p><img src="images/athenian_token_2.png" style="display:block;margin:auto;width:50%" alt="an Athenian voting token"></p>
<h1 id="participating-in-the-zeronet-rehearsal"><a class="toclink" href="#participating-in-the-zeronet-rehearsal">Participating in the Zeronet rehearsal</a></h1>
<p>In parallel, we have set up a rehearsal of the complete voting
procedure on the Zeronet test network. Zeronet moves at a faster pace
than Mainnet, giving the opportunity to try the voting procedure with
voting periods lasting only 51 hours (in the optimistic case where all blocks
are baked at the highest priority). If all goes well, the entire voting
procedure should have completed before the end of the first voting period on Mainnet.</p>
<p>The goal of this simulation is twofold.</p>
<ul>
<li>First, we want to test the complete procedure in a public
setting. This can help bring public confidence in the global process.
This is also the occasion for bakers who use third party or
homemade components (such as custom remote or hardware signers)
to test how their system behaves during automatic upgrades and on a test chain.</li>
<li>Second, this is the occasion for the community to learn to
communicate and debate during the vote. For this reason, we
encourage all participants to take the matter seriously. In
particular, we would not want to delay the procedure for too long
because someone abused the faucet and then forgot to launch their
baking daemon, ending up in lots of missed blocks. In the same
vein, we encourage all participants to be there and vote when their
participation is needed, so that we do reach the quorum and
supermajority, and the process can continue.</li>
</ul>
<p>Here is the current state of this rehearsal:</p>
<ul>
<li>On Tuesday, we reset the Zeronet, to set up the new time constants
that allow the vote to run much quicker than in Mainnet, while
still allowing people in different time zones to have a chance to
participate in each period.</li>
<li>It is now time for participants to deploy their nodes, use the
<a href="https://faucet.tzalpha.net/">faucet</a> to get some rolls, become
delegates and start baking.</li>
<li>Tomorrow, we shall publish a new Zeronet branch and Docker image that
embeds the two proposals (the same as in Mainnet), and the two sets
of associated baking daemons.</li>
<li>In the next few days, we shall inject the two proposals, launching
the voting procedure.</li>
</ul>
<p>We invite active participants to discuss and organise the rest of the
rehearsal in a specific channel <code>#zerovote</code> on the Baker’s slack
operated by Obsidian Systems (just email <code>tezos@obsidian.systems</code> for
an invite). And of course there is always
<a href="https://riot.im/app/#/room/#tezos:matrix.org">Riot</a> for open
discussions and <a href="https://tezos.stackexchange.com/">stackexchange</a> to
ask questions.</p>Athens: Our Proposals for the First Voted Amendment2019-02-25T16:00:00+01:002019-02-25T16:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2019-02-25:/athens-our-proposals-for-the-first-voted-amendment.html<p>Summary:</p>
<p>This blog post is a preview of <em>Athens</em>: our protocol proposal for the
first voted upgrade of Tezos.
As announced in <a href="https://research-development.nomadic-labs.com/meanwhile-at-nomadic-labs-1.html">the last <em>meanwhile at
Nomadic</em></a>,
we shall propose two upgrades: one lowers the roll size to 8,000 tez, the
other leaves it unchanged at 10,000 tez …</p><p>Summary:</p>
<p>This blog post is a preview of <em>Athens</em>: our protocol proposal for the
first voted upgrade of Tezos.
As announced in <a href="https://research-development.nomadic-labs.com/meanwhile-at-nomadic-labs-1.html">the last <em>meanwhile at
Nomadic</em></a>,
we shall propose two upgrades: one lowers the roll size to 8,000 tez, the
other leaves it unchanged at 10,000 tez.
Both alternatives will include an increase of the gas limit.</p>
<p><strong>The hashes of both versions will be proposed on mainnet later this week,
now that a new proposal period has begun.</strong></p>
<p><strong>Later this week, we will publish a blog post which provides the hashes and
instructions for voting, as well as optional instructions for bakers and
enthusiasts who want to participate in the tests.</strong></p>
<p><strong>Bakers who wish to upvote our proposals will have roughly three weeks
to do so.</strong></p>
<p>The same day we will also propose the same protocols on our experimental
Zeronet, where one voting period lasts 56 hours and the whole voting
procedure can be carried out in 9 days.
This gives us the opportunity to test the whole procedure and to
analyze the activation of a new protocol in Zeronet.</p>
<p>The actual Athens proposal is a self-contained set of OCaml sources. To make
it more readable, we have split it into a clean series of patches on
top of the current protocol. This article will describe each of these patches.</p>
<p>We recommend that users take the time to read the actual code and make
sure that it correctly reflects the advertised behavior.</p>
<p>For this first update, we decided to include an example of <em>invoicing</em>,
an on-chain way to fund the development of protocol proposals.
During the migration, the account of the authors of a proposal can be
funded by creating a certain amount of tez.
In this case, we chose a symbolic value of 100ꜩ, enough to buy a round of
drinks for the devs who worked on these proposals.
<strong>In the future, we hope this mechanism can be used to fund the work of
new teams and help increase decentralization.</strong></p>
<p>The next section presents a set of benchmarks for the gas and the
migration from protocol <code>Alpha_003</code> to <code>Athens_004</code>.</p>
<h1 id="building-confidence-in-a-protocol-amendment"><a class="toclink" href="#building-confidence-in-a-protocol-amendment">Building confidence in a protocol amendment</a></h1>
<p>Tezos has a built-in system to try new protocols: it can fork a <em>test
chain</em> from the main chain.
The test chain has the advantage of forking from an actual context of
mainnet, complete with all the accounts and contracts, thus giving a
realistic trial of what could happen at activation.</p>
<p>Additionally, it will be possible to try out the voting procedure and
the new protocol on the <em>Zeronet</em> development network.
This test will be public, it will run on a reduced timescale as cycles
are shorter in Zeronet, so it will be a good way to demonstrate the
whole process.
We will also inject our proposals on Zeronet, and while on
Mainnet the proposal period will last 3 weeks, on Zeronet it will take
just 56 hours.
If quorum and super-majority are reached, Zeronet could see a protocol
amendment in as little as 9 days.</p>
<p><strong>Prior to these large scale tests, we performed various tests and
benchmarks to make sure not only that the new protocol is behaving
correctly but also that the <em>migration</em> from the old protocol runs in
a reasonable amount of time and disk space.</strong></p>
<p>We will now give a short overview of three aspects that we thoroughly
tested before proposing the update: the gas limit update, the roll
size update, and the reindexation of accounts.</p>
<h2 id="gas-limits"><a class="toclink" href="#gas-limits">Gas Limits</a></h2>
<p>The new gas limits were decided by measuring the validation time of
worst case blocks (with a large number of computation intensive or <span class="caps">IO</span>
intensive contract executions), in a realistic mainnet context.</p>
<p>The cost of IOs was slightly underestimated, and thus the balance
between IOs and computation has been adjusted in this update. <strong>As a
result, the number of IOs performed in a block stays unchanged, while
the number of computation steps is doubled.</strong></p>
<p><strong>These new limits are kept within conservative bounds, allowing for a
smooth operation of the network.</strong></p>
<p>The Michelson benchmarking effort is continuing, in order to converge
to a more refined gas accounting scheme that should allow us to
propose a less conservative limit in a future upgrade.</p>
<p>The benchmarking method we are using will be the topic of a forthcoming series
of blog posts.</p>
<h2 id="roll-size-migration"><a class="toclink" href="#roll-size-migration">Roll size migration</a></h2>
<p>We simulated the upgrade in the current mainnet state, with and
without the roll size update, and measured the difference in
validation time and disk storage.</p>
<p>We tested two versions of the state, a full context of <span class="caps">72GB</span> and a snapshot generated
context of <span class="caps">330MB</span> (snapshots are a <a href="https://research-development.nomadic-labs.com/introducing-snapshots-and-history-modes-for-the-tezos-node.html">recently introduced
feature</a>).
In addition the migration was tested on Solid State Drives (<span class="caps">SSD</span>) and
Hard Disk Drives (<span class="caps">HDD</span>).</p>
<p>In a snapshot generated context, the roll size update increases the
migration time by less than 10 seconds on both <span class="caps">SSD</span> and <span class="caps">HDD</span>, while the
disk usage is increased by about <span class="caps">130MB</span>.
In a full context, the migration time is increased by 9.5 seconds on <span class="caps">SSD</span> and 26
seconds on <span class="caps">HDD</span>. The disk usage is increased by 340 <span class="caps">MB</span>.</p>
<p><strong>This cost only affects the migration block and has no impact on the
subsequent operation of the network.</strong></p>
<p><strong>In our opinion, the performance difference between the two protocol
proposals is not significant.</strong></p>
<p>The use of more efficient storage back-ends that we are currently
evaluating, shows even better performance in both time and disk usage.</p>
<h2 id="contract-reindexation"><a class="toclink" href="#contract-reindexation">Contract reindexation</a></h2>
<p>The new uniform indexing of contracts and accounts has no effect on
the behaviour of the protocol. It is a preliminary to future
improvements of the account system, such as making key handling more
flexible for bakers.</p>
<p>Migration from the current to the new internal structure presents a
noticeable cost in terms of time and disk that we carefully
benched to make sure that <strong>the overhead only affects the migration
block and has no impact on the subsequent operation of the network</strong>.</p>
<p>Using the current node implementation with a full mainnet context, the
migration takes about two minutes and increases disk occupancy by
around 5%. Running the same migration in a shallow
<a href="https://research-development.nomadic-labs.com/introducing-snapshots-and-history-modes-for-the-tezos-node.html">snapshot</a> reduces both time and disk usage by
an order of magnitude.</p>
<h2 id="a-note-on-disk-usage-and-migration-time"><a class="toclink" href="#a-note-on-disk-usage-and-migration-time">A note on disk usage and migration time</a></h2>
<p>As numbers above show, nodes bootstrapped from a snapshot, running
with partial archives, show much better performance. Snapshots are the
first step toward better storage that we want to integrate into the
node in the next few weeks. The preliminary tests of more efficient
experimental storage back-ends, that are the next step, show even
better performance in disk usage.</p>
<p>We kindly recommend that node administrators, in particular users of
magnetic disks for whom the migration will be heavier, give a try at
snapshots on a secondary machine, to help us speed up their
integration in the mainnet branch.</p>
<h1 id="changelog"><a class="toclink" href="#changelog">Changelog</a></h1>
<p>The full changelog of patches applied to the protocol is presented in
the following section.
The main changes consist in the invoicing, the changes to gas limits
and the change in rolls size which applies to one single proposal.
The remaining commits consist in improvements or corrections.</p>
<h2 id="main-changes"><a class="toclink" href="#main-changes">Main changes</a></h2>
<ul>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/3a39d70a4629f8629445bb159ad04630753d4e8a">Alpha/Michelson: double gas limit, except for I/Os that stay unchanged</a></p>
<p>This change doubles the maximum amount of gas for the block and for
each transaction. The cost of I/Os is updated accordingly, so that the
number of computation steps is increased, while the number of possible
I/Os in a block stays unchanged.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/d65c58c24dd8de4fbe4385297ca0092a4841c347">Alpha/Michelson/Migration: double hard_gas_limit_per_{operation,block}</a></p>
<p>This patch is part of the migration code from <code>alpha_003</code> to <code>athens_004</code>.</p>
<p>It performs the migration of the constants in mainnet to update them
to the new (doubled) default values.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/5e2b979054ee5dde42c427f8915cb2e027f6d234">Alpha/Delegation: set the tokens_per_roll constant to 8000 tez</a></p>
<p>This patch sets the new default roll size to 8000 tez.</p>
<p><strong>only in one proposal</strong></p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/b9408be70c4e6643f6787c82f743ae10d9045c17">Alpha/Delegation/Migration: reassign rolls</a></p>
<p>This patch is part of the migration code from <code>alpha_003</code> to <code>athens_004</code>.</p>
<p>It performs the roll size update to 8000 tez.</p>
<p><strong>only in one proposal</strong></p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/e010d7007b85f1fa33f9ff6511b4fbc12706d048">Alpha/Migration: invoicing example</a></p>
<p>This patch is part of the migration code from <code>alpha_003</code> to <code>athens_004</code>.</p>
<p>It credits the account of the authors of this protocol with a symbolic
value of a hundred tez.</p>
<p>The authors hereby declare that these tez will be used exclusively
to pay for a round of drinks for the Nomadic team.</p>
</li>
</ul>
<h2 id="other-changes"><a class="toclink" href="#other-changes">Other changes</a></h2>
<ul>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/dcec044294e072a7c57e273fc7d80d28eee57b84">Alpha/Refactor: remove unused compare module in constants</a></p>
<p>This commit removes a small piece of unused code.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/e8213708de33c08ad65898c6793a4f1c8d6853a7">Alpha/Refactor: use return_*</a></p>
<p>Replaces some allocations of common return values by preallocated constants.
This is a cosmetic change that does not alter the semantics.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/a3d04dd2ef157c7ffeb1d5a0c4b056d361ccbb5e">Alpha/Refactor: simplify code of <code>Raw_context.prepare_first_block</code></a></p>
<p>This patch simplifies the protocol migration code.
It does not change the semantics.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/5b41069384f18382c09bc9c967645f61d4427f5f">Alpha/RPCs: fix wrong encoding name fields</a></p>
<p>This patch fixes a mislabeled field in the <span class="caps">JSON</span> encoding of operation receipts.
The balance updates had a field <code>level</code> that actually contained a cycle.
It is now named <code>cycle</code>.</p>
<p>Third party software that make use of receipts such as block explorers
or wallets may need to be updated accordingly. A simple solution is to
support both during the protocol change period.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/50cf26f012c456248a2a01d59e654fe9993b117b">Alpha/RPCs: change <code>managerPubkey</code> to underscores in json field names</a></p>
<p>This renames a field in the <span class="caps">JSON</span> format of origination operations to
make it consistent with the others.</p>
<p>Third party software must be updated to support this change.
For applications that just consume this format, it is enough to just recognise
both field names during the protocol change period. For applications that send
this format to the node, it is necessary to send the right one depending on the
<code>next_protocol</code> field of the interrogated block.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/7b4fe1b9fbaae07efccf996b7233e83768e9bfce">Alpha/RPCs: rename field labels in Michelson typechecker output</a></p>
<p>This is a change in field names in the Michelson typechecking <span class="caps">RPC</span> output.
A simple solution for third party tools to handle the change is to
recognise both field names during the protocol update period.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/94cfcfd21d2fb31c549beb360c7bc2470c1d8c35">Alpha/RPCs: rename Michelson interpreter diagnostics field names</a></p>
<p>This patch changes some error IDs and error attributes to make it
consistent with the rest of the codebase.</p>
<p>Third party development tools that recognise errors based on this
should be updated.As errors are only read, a simple solution is to
support both during the protocol change period.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/2d51e9965858cb3026a249bc0ec0614b9b36a2bb">Alpha/RPCs: optional fields for source, payer, gas in run and trace</a></p>
<p>This patch adds new options to the Michelson interpreter RPCs.
They allow customization of interpreter behaviour.
It only makes utility RPCs more flexible, and does not change the
semantics of Michelson.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/91fa2d7fb7012901950e547eee6cde301d850d95">Alpha/Vote: fix initialization of voting rights from genesis</a></p>
<p>This patch fixes the voting system initialisation from genesis.</p>
<p>It improves voting in sandbox mode and test networks.
It has no effect on mainnet.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/09794e40c5c8ab95b578303ff946b22f4872ac55">Alpha/Vote: docstrings and one renaming</a></p>
<p>This patch adds documentation to the code.
It does not change the semantics.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/d98fe4e5c9c4418f00d409fcabdcefd6f5a72cb8">Alpha/Vote: fix potential overflow in quorum computation</a></p>
<p>This patch prevents a potential overflow if the number of rolls gets
too big in a future update. This bug cannot happen with the current value.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/34aaf6c7c353c24c1c3420618cf9c1c357e300e6">Alpha/Vote: new constant <code>test_chain_duration</code></a></p>
<p>This patch turns the test chain lifetime from a hardcoded two-days
constant in the code to a parameter. This is useful for testing of
the voting system, for testnet rehearsals, or for more easily changing
the value in a future protocol upgrade.</p>
<p>The value of this constant is also updated to prolong the test chain
to the end of period. The test chain is now spawned at the beginning of
the third voting period and lasts the whole period.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/d0c16b4ff8439764dad5053f3af607e1ca44ca1c">Alpha/Vote/Migration: update constants binary representation</a></p>
<p>This patch is part of the migration code from <code>alpha_003</code> to <code>athens_004</code>.</p>
<p>Since we changed <code>Constants_repr.parametric_encoding</code> for adding the
new constant <code>test_chain_duration</code>, we need to upgrade the content
of the <code>constants_key</code> in the context.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/f6d67b7afc5b3c3622941fdb62ca319546e736a8">Alpha/Michelson: allows big maps in genesis blocks</a></p>
<p>This patch allows to create a sandbox chain with bootstrap contracts
that use big_maps. This change is mostly useful for testing in
sandbox mode or custom deployments.</p>
<p>This has no effect on mainnet.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/01b5bb080bd6eb6b33b8654526785b20c403fc6a">Alpha/Michelson: do not allow spendable smart contracts</a></p>
<p>This patches forbids the creation of spendable smart contracts, a
current feature that is not necessary (it can be implemented from the
contract’s code) and prevents some possible future optimisations.</p>
<p>It also prevents the creation of non spendable non scripted contracts,
which was mostly a design flaw that could let people inadvertantly lock funds.</p>
<p>This is done by checking the flag in the origination operation rather
than removing it so the change is lighter for third party software.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/a96b03dd729a4efbbce81b06d213fcd99d09bc45">Alpha/Delegation: do not activate delegate when crediting</a></p>
<p>Remove the unneeded method for a delegate to reactivate itself by
provisioning its account. The delegate should simply re-register as delegate.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/079eda15b2f41de6d5e2cc768d22a7770485245c">Alpha/Delegation: allow to change the roll value</a></p>
<p>This patch adds a function to reassign rolls given a new roll size.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/82fc0334c2f384f99f6630976627506e82c1f9ca">Alpha/Contracts: uniform indexation for both implicit and originated contracts</a></p>
<p>This patch change the way contracts are indexed internally, using the
same database paths for implicit contracts, delegate accounts,
scripted and non scripted contracts.</p>
<p>It simplifies future updates, for instance to add new account types.</p>
</li>
<li>
<p><a href="https://gitlab.com/nomadic-labs/tezos/commit/8618b4754c80720062056320451abc7338605631">Alpha/Contracts/Migration: reindex contracts</a></p>
<p>This patch is part of the migration code from <code>alpha_003</code> to <code>athens_004</code>.</p>
<p>It performs the migration of contracts to the new uniform storage format.</p>
</li>
</ul>Amendments at Work in Tezos2019-02-13T18:00:00+01:002019-02-13T18:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2019-02-13:/amendments-at-work-in-tezos.html<p>Summary:</p>
<p>We are now on the verge of submitting a protocol upgrade to a vote,
and it seems like a good opportunity to explain in details the way in
which Tezos node handles amendment in practice.</p>
<p>Brace yourselves, this article is quite technical, as are all articles in
our <em>in-depth …</em></p><p>Summary:</p>
<p>We are now on the verge of submitting a protocol upgrade to a vote,
and it seems like a good opportunity to explain in details the way in
which Tezos node handles amendment in practice.</p>
<p>Brace yourselves, this article is quite technical, as are all articles in
our <em>in-depth</em> category. Still, as we did in the previous one on
snapshots, we’ll try to explain the stakes and announcements and give
a brief summary in a short foreword understandable even by non-programmers.</p>
<p>The original whitepaper by <span class="caps">L.M.</span> Goodman describes Tezos as a system
capable of engulfing any blockchain protocol. A blockchain is
ultimately a ledger whose state (the balances and stakes) is
transformed by operations, and a consensus algorithm, which chooses
among all possible alternative heads proposed by the network.
They are modelled by two functions, in the mathematical sense,
<span class="math">\(apply\)</span> and <span class="math">\(fitness\)</span>. From an initial state <span class="math">\(S_0\)</span>, <span class="math">\(apply(B_n,S_{n-1})\)</span>
is called at each block of height <span class="math">\(n\)</span> to produce the current state of the
ledger <span class="math">\(S_n\)</span> (called context in the codebase). <span class="math">\(fitness(S_n)\)</span> gives a score to
each alternative chain to decide which chain is the current chosen head.
The plot twist is that this new state <span class="math">\(S_n\)</span> can actually contain two new
functions <span class="math">\(apply\)</span> and <span class="math">\(fitness\)</span>, that will replace the current ones for the next block.</p>
<p>This neat mathematical abstraction is the very essence of Tezos. In
this article, we try to demystify how this concept is turned into an
actual evolving blockchain that can amend itself to remain at the state of the
art and learn from its comrades’ (i.e. other protocols’) innovations and mistakes.</p>
<p>The software architecture of Tezos has often been illustrated by the
following image, where the node is represented by a colorful octopus (the famous
<em>El Pulpo</em>).</p>
<p><img alt="visual depiction of Tezos architecture" src="images/octopus.svg"></p>
<p>The octopus interacts with the rest of the network via its arms,
stores the data in its belly, and uses its interchangeable brain to
validate the chain and select its head.</p>
<p>The take away of this drawing is that the
Tezos node is split into two zones, one containing the other. The node
is mostly a <em>shell</em>, a generic blockchain network and storage layer,
that exchanges blocks and operations with its peers, without knowing
much about their content. It sees a tree of possible chains, and has
to decide which one is considered the head.</p>
<p>To make sense out of all this data coming from the network, the shell
relies on <em>protocols</em> as black boxes. In order to choose its
head, the shell will look at all its possible candidate heads, find
the protocol associated to each one, call their <span class="math">\(fitness\)</span> function,
and select the one with the higher score. For instance, in the current protocol,
this score is the number of endorsements and blocks in the chain.
To validate blocks and operations, the protocol relies on its current
state (context). For the shell, this is an append-only database
storing arbitrary data. It is in the protocol’s purview to give sense
to its content.</p>
<p>The first section of this article describes in more technical terms
what these black boxes contain, the format they have to respect, and
how they interact with the shell.</p>
<p>During a Tezos amendment, when a protocol or a user decides that it is time
for a protocol change, the shell has to retrieve and switch to the new
protocol. This protocol has an additional migration function <span class="math">\(init\)</span>
that knows how to convert the content of a previous protocol to its
own format.</p>
<p>The next two sections explain what happens in practice during an
amendment, and the final section gives a more concrete vision with a survey of
past protocol upgrades. In other words, a brief history of Tezos.</p>
<h2 id="the-protocols-who-are-they-where-do-they-live-to-whom-do-they-answer"><a class="toclink" href="#the-protocols-who-are-they-where-do-they-live-to-whom-do-they-answer">The protocols: who are they? where do they live? to whom do they answer?</a></h2>
<p>Practically speaking, a protocol is a series of OCaml modules
definitions and interfaces that must respect a set of rules to make
sure that they can be loaded properly by the Tezos shell.</p>
<h3 id="protocol-format"><a class="toclink" href="#protocol-format">Protocol format</a></h3>
<p>There are two reference formats for Tezos protocols. The on-disk
format, input of the Tezos compiler, is a folder containing <code>.ml</code> (source code) and
<code>.mli</code> (signature) files, accompanied by a <code>TEZOS_PROTOCOL</code> json description, that
contains the list of module in compilation order, an optional
environment version (see next section), and an optional protocol hash
override for testing purposes. The packed format is a binary encoding
described in <code>src/lib_base/protocol.ml</code>. This second format is the one
used to compute the protocol hash and to transmit protocols over the
peer-to-peer network.</p>
<p>The module list must end in a module named <code>Main</code> that contains the
definitions of <span class="math">\(apply\)</span>, <span class="math">\(fitness\)</span> and <span class="math">\(init\)</span>. In reality, they are
broken down into smaller data types and functions, and are accompanied
by a few utilities such as data encodings and <span class="caps">RPC</span> definitions. The
exact signature that <code>Main</code> must respect is called <code>PROTOCOL</code>, and is
defined in <code>src/lib_protocol_environment/sigs/v1/updater.mli</code>. This is
how the shell interacts with protocols in a standardised and type-safe
way, considering them as black boxes that all have the same input
plugs. Basically, you can see the protocols as plug-ins that conform
to a common plug-in interface.</p>
<h3 id="the-protocol-environment"><a class="toclink" href="#the-protocol-environment">The protocol environment</a></h3>
<p>Continuing on this (relevant) analogy with plug-ins, protocols
have access, for convenience, to a library of functions
(including cryptographic primitives) and
access to the state (such as the stored balances) in a way that is safe,
and maintainable. In other words, they have access to a stable plug-in
<span class="caps">API</span>.</p>
<p>For that, the OCaml code of a protocol is not compiled and run in the
same environment as the rest of the node. Instead, it can only access the
modules whose interfaces are given in <code>src/lib_protocol_environment/sigs/v1/</code>.
These modules cover standard OCaml and Tezos-specific libraries that have been
stripped down to remove any unsafe or unmaintainable function. This is a form of
sandboxing achieved through the OCaml type-system.</p>
<p>When the shell loads a protocol, it provides an actual implementation
for each module in the environment. As a concrete example, the
<code>begin_application</code> function from the <code>PROTOCOL</code> interface takes a
value of type <code>Context.t</code>, that is the database that contains the
ledger’s state. The protocol only knows that this type exists and has
a few associated functions. When a protocol is loaded, this type and
functions are plugged to the actual on-disk storage primitives.</p>
<p>This abstraction is not just interesting for reducing the attack
surface via sandboxing, it also open possibilities such as replacing
the implementation of functions by newer, better ones, implementing
different shells, etc. For instance, the test framework of Tezos uses
a variant that runs entirely in <span class="caps">RAM</span>.</p>
<h3 id="different-clients-for-different-protocols"><a class="toclink" href="#different-clients-for-different-protocols">Different clients for different protocols</a></h3>
<p>If you look at the source code of Tezos mainnet, you will see the code
for all the previous protocol upgrades in folders of the form
<code>src/proto_<index>_<hash>/</code>. Each of these folders contains a subfolder
<code>lib_protocol/src/</code> which is the actual code of the protocol, but also
others, such as <code>bin_baker</code> of <code>lib_client_commands</code>.</p>
<p>For running a node, only the code in
<code>lib_protocol/src/</code> is necessary, and it is not even necessary to have it locally,
since the shell could request it from the peer to peer network. But to use
this protocol from the command line client, or to run a baker for this
specific protocol, the other pieces are needed, and cannot be
downloaded from the network.</p>
<p>Indeed, the important part, the one people agree upon when they vote, is
the protocol itself. The command line client is just one client among
others, and the baker, endorser and accuser are just reference
implementations that we’ve provided, but others can write their own
custom versions. It would not make sense either to provide generic
implementations, as Tezos is flexible enough to completely change the
consensus algorithm or the account system, and thus the code that
builds blocks or produces transactions must be specific to each protocol.</p>
<p>Running a baker and running a node to participate in the system
represent different levels of involvement. A baker has to understand and
prepare for protocol changes, because they will have to run a new baker and,
depending on the update, may have to adapt to modifications in their setup. </p>
<p>But if you just run a node, you can launch it once, and updates will come to
you safely by themselves as they are proposed and approved by Tezos stakeholders.</p>
<h2 id="amendment-blocks"><a class="toclink" href="#amendment-blocks">Amendment blocks</a></h2>
<p>To trigger a protocol change — for instance, in the current system,
at the end of a successful voting cycle — the protocol has access in its
environment to a <code>Updater.activate</code> function. This function will write the hash
of the chosen protocol in a specific place in the context.</p>
<p>In that case, the shell, after validating the block, will read the hash,
and look for the protocol. It will at this point download it from
its peers, compile it and load it as a plug-in.</p>
<p>The next step is then to prepare the context format for the next
protocol by calling its <span class="math">\(init\)</span> function. This function will initialise
any new structure introduced. For instance, if shielded votes or
transactions are introduced, they may require new structures in the
context. It may also migrate existing structures if needed. For
instance, the upcoming proposal will include a transition in the structure
of accounts, as a first cleanup to prepare for a forthcoming rehaul of
the account system. We informally call that practice <em>stitching</em> the context.</p>
<p>If you consult such a transition block via the node’s RPCs, you will
notice that it has two fields <code>protocol</code> and <code>next_protocol</code>. The
former is the protocol that parsed and validated the block, providing
a clear <span class="caps">JSON</span> version of the block contents
(header, operations and receipts). The latter is the one that can run
in the newly migrated context, defining the contextual RPCs (i.e RPCs being
protocol dependent, such as getting an account’s balance).
The first example in the third part gives a more concrete views of these two fields.</p>
<h2 id="automatic-vs-user-activated-upgrades"><a class="toclink" href="#automatic-vs-user-activated-upgrades">Automatic vs. user-activated upgrades</a></h2>
<p>Tezos has integrated support for two kind of upgrades: automatic
upgrades, and user-activated upgrades.</p>
<p>Automatic upgrade is the method that we just explained in details, and
that starts when the protocol decides to call <code>activate</code>. In the
current protocol of mainnet, this <code>activate</code> function is called only on a
protocol that passed a successful voting procedure. But to be precise
this voting procedure is a specific case of a more general
procedure, and not carved in stone at all. Using this voting
mechanism, people could vote to switch to a different voting system,
or to a dictatorship. To be even more dramatic, people could decide to
vote for a protocol that never calls <code>activate</code> anymore, forcing the
Tezos mainnet to use the same protocol for eternity.</p>
<p>User-activated protocol upgrades have different purposes. The main use
case is non controversial bug fixes or emergency reactions to
unexpected behaviour. Of course, people could also try and use that to
instigate a revolution and bypass the automatic upgrade system (for instance,
to correct the silly situation described at the end of the last
paragraph). All of this you can find in great details in <a href="https://medium.com/tezos/there-is-no-need-for-hard-forks-86b68165e67d">this
article</a>
by Arthur Breitman.</p>
<h2 id="a-quick-history-of-past-upgrades"><a class="toclink" href="#a-quick-history-of-past-upgrades">A quick history of past upgrades</a></h2>
<p>As a concrete example of how the amendment works in practice, let us
review the history of Tezos protocol upgrades to date.</p>
<h3 id="from-proto_000_ps9mpmxa-genesis-to-proto_001_ptcj7pwo-alpha"><a class="toclink" href="#from-proto_000_ps9mpmxa-genesis-to-proto_001_ptcj7pwo-alpha">From <code>proto_000_Ps9mPmXa</code> (<code>"genesis"</code>) to <code>proto_001_PtCJ7pwo</code> (<code>"alpha"</code>)</a></h3>
<p>This is what happened on June 30th of last year. Some early birds
where already running nodes in the network, waiting for the activation block.</p>
<p>Technically, when a Tezos mainnet node boots up from an empty data
directory, if forges a dummy block (the same for everyone) at level
zero. This block has no data, its identifier is
<code>BLockGene...sisf79b5d1CoW2</code>, and its <code>next_protocol</code> is set to
<code>proto_000_Ps9mPmXa</code>. This means that any block at level one will be
parsed and validated by this <code>"genesis"</code> protocol.</p>
<p>The genesis protocol accepts a single block (at level one) that
contains a protocol hash, and data to pass to its initialisation
function placed at a specific place in the context. In the case of the mainnet
(betanet at the time), the activated protocol was
<code>proto_001_PtCJ7pwo</code>, codename <code>"alpha"</code>, and the data was a list of
wallet allocations. The node will then call the initialization
function of this new protocol, so that the context is prepared
(initialized with the initial balances of the wallets) for protocol
<code>"alpha"</code> to validate the subsequent blocks properly.</p>
<p>Let us take this example to illustrate the difference between the
<code>protocol</code> and <code>next_protocol</code> fields in the <span class="caps">RPC</span> <span class="caps">API</span> explained
earlier. This activation block at level 1 has its field <code>protocol</code> set
to <code>proto_000_Ps9mPmXa</code>: the protocol who parses and validates the
block. Its <code>next_protocol</code> field is however <code>proto_001_PtCJ7pwo</code>: the
protocol who responds to <span class="caps">API</span> calls and will evaluate the next block.</p>
<p>In a sandbox, or in a custom deployment, you can push your own
activation block, and launch <code>"alpha"</code> with your own set of
wallets. Actually, the Tezos node is not even bound to start
<code>"alpha"</code>, you could start any protocol. For instance, the test
network Alphanet and Zeronet goes directly from <code>"genesis"</code> to the last
update when they are rebooted.</p>
<p>This is a first example of an automatic upgrade. Nodes will receive a
block, and the protocol will decide to amend itself during its
evaluation. Then, nodes will download a copy of the protocol from the
network if they don’t have it built-in, and continue receiving and
evaluating blocks using this new protocol.</p>
<h3 id="from-proto_001_ptcj7pwo-alpha-to-proto_002_psylvpvv-alpha_002"><a class="toclink" href="#from-proto_001_ptcj7pwo-alpha-to-proto_002_psylvpvv-alpha_002">From <code>proto_001_PtCJ7pwo</code> (<code>"alpha"</code>) to <code>proto_002_PsYLVpVv</code> (<code>"alpha_002"</code>)</a></h3>
<p>This migration was a user-activated upgrade at level 28082. At this
level, <code>proto_001_PtCJ7pwo</code> did not perform an activation. However,
the development team at this time suggested to migrate
manually to a minor revision of the protocol, and node administrators
who chose to follow this user-activated branch had to force their node
into performing a protocol upgrade at this level (editing the source
manually, or pulling from the repository).</p>
<p>This upgrade proposal was meant to fix a few bugs that made their way
into the betanet launch:</p>
<ul>
<li>The origination of a new smart contract from within a smart
contract was spending the initial balance provision twice.</li>
<li>Serialisation of Michelson bytecode was not implemented,
preventing the use of instruction <code>LAMBDA</code>.</li>
<li>Using a specific method, it was possible to delegate funds to an
account that did not register themselves as delegate.</li>
</ul>
<p>This upgrade also introduced a few Michelson instructions to work on a
generic <code>bytes</code> type when releasing <code>proto_001_PtCJ7pwo</code>.</p>
<h3 id="from-proto_002_psylvpvv-alpha_002-to-proto_003_psddfki3-alpha_003"><a class="toclink" href="#from-proto_002_psylvpvv-alpha_002-to-proto_003_psddfki3-alpha_003">From <code>proto_002_PsYLVpVv</code> (<code>"alpha_002"</code>) to <code>proto_003_PsddFKi3</code> (<code>"alpha_003"</code>)</a></h3>
<p>This migration was the second user-activated upgrade at
level 204761. Here again, it was a bugfix release, and node
administrators who chose to follow this user activated branch had to
activate it knowingly. However, the community at this time was already
much bigger, and it required time to coordinate. Hopefully, as Tezos
matures and formal methods are applied to more parts of the codebase,
user-activated releases will not be necessary, and all updates can happen
smoothly via self-amendment.</p>
<p>At that time, an upgrade proposal was already on our shelves. Its goal
was to fix what needed to be fixed in the protocol to allow for the
first vote. Indeed, when launching the betanet, we concentrated on
having strong consensus and transaction layers in the protocol, but we gave less
love to the self-amendment part. Hence, it was not unexpected that a few fixes
here and there would have to be implemented using a user-activated upgrade.
These fixes included adding missing RPCs and correcting the ballot counts.</p>
<p>Our timeline for proposing this upgrade was shuffled a bit by an
increased level of noise, or spam, on the network in November, due to some
entity creating a lot of adresses with very low balance, and making many useless
transactions of microtez amounts between them.</p>
<p>This spamming was possible because address creation was free, on
purpose, to encourage adoption and barrierless entry to the
system. This abuse led us to apply the same storage
cost policy to adresses as to originated contracts.</p>
<p>Another spam prevention method, popular amongst bakers, was introduced.
The network was running fee-less at the time. As part of the upgrade, the shell
was modified to impose fee thresholds in the mempools, and a similar mechanism
in the baker to include only transactions that respect a given fee scheme. This
later change was not a change in the protocol itself but piggy-backed on top
of the coordination achieved for the user-activated upgrade.</p>
<h3 id="from-proto_003_psddfki3-alpha_003-to-athens"><a class="toclink" href="#from-proto_003_psddfki3-alpha_003-to-athens">From <code>proto_003_PsddFKi3</code> (<code>"alpha_003"</code>) to <code>???</code> (<code>"Athens"</code>)</a></h3>
<p>As we explained in the last <em>meanwhile at Nomadic</em>, our upcoming upgrade
proposal(s) will contain not only fixes, but actual protocol changes
that require a vote by the Tezos community. We will detail all
proposed changes and alternatives in a forthcoming post. </p>
<p>For now, let us describe a small but defining change in our upcoming proposal.
Up to now, protocol names (quoted in the titles above, such as the
current <code>"alpha_003"</code>) have been mere identifiers, stored in a
specific place of the context to identify the current version.</p>
<p>We want to use the first voted proposal as an opportunity to
adopt a new policy, and instead use these version strings to give a
proper name to each new Tezos protocol version.</p>
<p>Our suggestion is to use city names, anglicised and in alphabetic
order. Of course, in the future other entities will propose
protocol upgrades, and it will be up to them to follow the convention
or not. Yet, city names provide a wide set to choose from for each
letter, with even a bit of room to express things. </p>
<p><strong>As a demonstration, <code>"Athens"</code> is a pretty obvious choice for this first voted
upgrade, don’t you think?</strong></p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Introducing Snapshots and History Modes for the Tezos Node2019-02-04T18:00:00+01:002019-02-04T18:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2019-02-04:/introducing-snapshots-and-history-modes-for-the-tezos-node.html<p>In this article, we introduce two new features for the Tezos node:
snapshots and history modes.</p>
<p>A snapshot is a file that contains everything necessary to
restore the state of a node at a given block. A node restored via a snapshot
can synchronise and help other nodes synchronise in the existing
network. The only difference is that you cannot query the chain
context (balances, baking rights, etc.) before the restoration point,
but you can still get the full chain history.</p>
<p>In conjunction, we also introduce history modes, which represent different
policies for determining which past data a node should maintain. We propose
three modes: <code>archive</code> (the current mode which keeps everything), <code>full</code>
(the new default) and <code>rolling</code>. For now, snapshots can fire up a node
in either <code>full</code> or <code>rolling</code> mode.</p>
<p>These new features allow a user to spawn and synchronise a Tezos node in a
few minutes, from a single, untrusted file of about <span class="caps">150MB</span> compressed
with a truncated history, or <span class="caps">800MB</span> with a full history. You can test
all of that by using the <code>mainnet-snapshots</code> branch on Nomadic Labs’ Gitlab.</p>
<p>Be aware that this is not yet production ready, it would not be wise
to replace your current infrastructure with nodes from this branch at
this date. However, you are very welcome to experiment with it, and
all reports will be useful if we want this feature to be merged in
mainnet as soon as possible.</p>
<h2 id="history-modes"><a class="toclink" href="#history-modes">History modes</a></h2>
<p>History modes allow the node to run without maintaining the full
archives of the chain.</p>
<p>Here are the three first modes:</p>
<ul>
<li><code>full</code> nodes store all chain data since the beginning
of the chain, but drop the archived contexts below the current
checkpoint. In other words, you can still query any block or
operation at any point in the chain, but you cannot query the
balances or staking rights too far in the past.</li>
<li><code>rolling</code> nodes are currently the most lightweight, only
keeping a minimal rolling fragment of the chain and deleting
everything before this fragment (blocks, operations and archived contexts).</li>
<li><code>archive</code> nodes store everything. This corresponds to the
current behaviour of Tezos nodes.</li>
</ul>
<p>Full nodes will be the new default, as they are sufficient for almost
everyone. We plan to introduce new modes in the future.</p>
<p>An important thing to note is that running a full node is enough to
maintain the full chain history. Indeed, archive nodes do not need
to use archive peers to bootstrap their archive, but only full peers, as the
chain data is enough to apply the chain and construct the context
archives. In other words, the network does not lose any security by
switching to full as the default.</p>
<h3 id="checkpoints"><a class="toclink" href="#checkpoints">Checkpoints</a></h3>
<p>To understand the technical details of history modes, let us first
recall what <em>checkpoints</em> are in Tezos.</p>
<p>In the current protocol, an automatic checkpoint is <em>cemented</em> by the
node to the block at position 0 of the 5th previous cycle.</p>
<p>The aim of checkpoints is to anchor the consensus in the real world at regular intervals. The block hashes from this point can
easily be shared, propagated and saved outside the chain. You may have
already heard of them since checkpoints were mentioned in the <a href="https://tezos.com/static/papers/position_paper.pdf#14">Tezos
position
paper</a>.</p>
<p>When the checkpoint (<span class="caps">CP</span> in the picture below) is updated (currently at
each cycle), alternative branches that do not contain the
checkpoint (i.e those that diverged from the main chain before the new
checkpoint’s level) are marked as invalid and can be safely
deleted. The node does not accept reorganisations below this point.</p>
<p><img alt="visual representation of the checkpoint with discarded old branches" src="images/snapshots/checkpoint.svg"></p>
<p>A new <span class="caps">RPC</span> is available in order to request the current checkpoint (as
well as additional new information) of a chain.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos<span class="w"> </span>rpc<span class="w"> </span>get<span class="w"> </span>/chains/main/checkpoint
</code></pre></div>
<h3 id="full-mode-the-new-default"><a class="toclink" href="#full-mode-the-new-default">Full mode, the new default</a></h3>
<p>The <code>full</code>-mode is the default mode when starting a node from scratch, or
from a full snapshot.</p>
<p>A node running in <code>full</code>-mode stores the full chain data for all
blocks, even the ones older than the current checkpoint. More
precisely, it keeps the <em>headers</em> and the <em>operations</em> for these
blocks. However, it discards the archived <em>context</em> and the
operation and block <em>receipts</em>. We say that such a block
information is “pruned”: we keep only the necessary bits that we got
from the network, and drop everything that can be reconstructed from them.</p>
<p>In practice, we introduce two new tagged blocks in the history: the
<em>save point</em> and the <em>caboose</em>. The save point currently
mirrors the checkpoint and references the oldest block that contains
all the data, i.e the oldest one that is not pruned. The caboose
corresponds to the oldest pruned block. Here is a picture illustrating
the full-mode initialisation.</p>
<p><img alt="full node history at initialisation" src="images/snapshots/full0.svg"></p>
<p>The save point (<span class="caps">SP</span>) is first initialized with the checkpoint (<span class="caps">CP</span>)
referenced by the snapshot, and the caboose (<span class="caps">OO</span>) with the oldest
pruned block included by the snapshot.</p>
<p>Each time the chain checkpoint is updated, we also update the save
point and the blocks older than the new save point are pruned.
Finally, the caboose stays unchanged. Here is a picture
illustrating the state of the chain in full-mode after a checkpoint update.</p>
<p><img alt="full node history at a checkpoint update" src="images/snapshots/full1.svg"></p>
<h4 id="how-to-use"><a class="toclink" href="#how-to-use">How to use</a></h4>
<p>Using <code>tezos-run</code> command line arguments:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-node<span class="w"> </span>run<span class="w"> </span>--history-mode<span class="w"> </span>full
</code></pre></div>
<p>Or the configuration file:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"> </span><span class="nt">"shell"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nt">"history_mode"</span><span class="p">:</span><span class="w"> </span><span class="s2">"full"</span><span class="p">}</span><span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>If you start your node for the first time, this argument is not
necessary as it is now the default. However, if you upgrade from an
existing archive state and want to switch to full mode, you can pass
the argument to convert your <code>archive</code> node to a <code>full</code> one.</p>
<h3 id="rolling-mode-the-lightest"><a class="toclink" href="#rolling-mode-the-lightest">Rolling mode, the lightest</a></h3>
<p>The rolling-mode is the lightest mode for now.
It only conserves a pruned history for a minimal period of blocks before
the current save point.
The difference with the full-mode is that a rolling node also updates
the caboose and deletes blocks that are older than this one.</p>
<p>When starting a node configured in rolling-mode, the caboose and the save point
are initialized the same way as for the full-mode.
Here is a picture illustrating the rolling-mode initialisation.</p>
<p><img alt="rolling node history at initialisation" src="images/snapshots/rolling0.svg"></p>
<p>Whenever the current checkpoint is updated, the node will also update its
caboose and its save point in such a way that the distance between the new
save point and the new caboose corresponds to the lifetime of operations
(required to ensure proper validation of reorganisations just after the checkpoint).
It will then purge its store by deleting all block information for those
older than the new caboose and pruning all blocks between the new
caboose (included) and the save point (excluded).
Here is a picture illustrating the state of the chain in rolling-mode after
a checkpoint update.</p>
<p><img alt="rolling node history during a checkpoint update" src="images/snapshots/rolling1.svg"></p>
<h4 id="how-to-use_1"><a class="toclink" href="#how-to-use_1">How to use</a></h4>
<p>Using <code>tezos-run</code> command line arguments:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-node<span class="w"> </span>run<span class="w"> </span>--history-mode<span class="w"> </span>rolling
</code></pre></div>
<p>Or the configuration file:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"> </span><span class="nt">"shell"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nt">"history_mode"</span><span class="p">:</span><span class="w"> </span><span class="s2">"rolling"</span><span class="p">}</span><span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>In that mode, the new checkpoint <span class="caps">RPC</span> will also give you the save point
and caboose.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos<span class="w"> </span>rpc<span class="w"> </span>get<span class="w"> </span>/chains/main/checkpoint
</code></pre></div>
<h3 id="archive-mode"><a class="toclink" href="#archive-mode">Archive mode</a></h3>
<p>The archive mode aims to save all the chain data, starting necessarily at the genesis block.
It corresponds to the one the nodes are currently using.
The archive mode can be useful in the context of
block explorers for instance.</p>
<h4 id="how-to-use_2"><a class="toclink" href="#how-to-use_2">How to use</a></h4>
<p>Using <code>tezos-run</code> command line arguments:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-node<span class="w"> </span>run<span class="w"> </span>--history-mode<span class="w"> </span>archive
</code></pre></div>
<p>Or the configuration file:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span><span class="w"> </span><span class="nt">"shell"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nt">"history_mode"</span><span class="p">:</span><span class="w"> </span><span class="s2">"archive"</span><span class="p">}</span><span class="w"> </span><span class="p">}</span>
</code></pre></div>
<p>If you want to start an <code>archive</code> node, it is now mandatory to pass
this argument the <strong>first</strong> time you launch your node.</p>
<h3 id="from-a-mode-to-another"><a class="toclink" href="#from-a-mode-to-another">From a mode to another</a></h3>
<p>There are some restrictions when one wants to switch from a mode to another.</p>
<p>Going from <code>archive</code> to <code>full</code> or <code>rolling</code> or from <code>full</code> to <code>rolling</code>
is allowed, as it is just dropping data. It is not allowed to switch
from the <code>full</code> or <code>rolling</code> to <code>archive</code>, since the last one would
require to rebuild dropped archives.</p>
<p>We have plan to leverage that restrictions in the future.</p>
<h2 id="snapshots"><a class="toclink" href="#snapshots">Snapshots</a></h2>
<p>As the chain invariably grows every day, retrieving a full chain from
the peer-to-peer network can be a very long process. Thanks to the
implementation of history modes, it is now possible to propose an
import/export feature: snapshots. This procedure allows to gather all
the data necessary to bootstrap a node from a single file.</p>
<h3 id="starting-a-node-from-a-snapshot"><a class="toclink" href="#starting-a-node-from-a-snapshot">Starting a node from a snapshot</a></h3>
<p>When bootstrapping from a snapshot, the first thing that you want to
do is check the point in history from when you start.</p>
<p>The snapshot format does not (and cannot) provide any evidence that
the imported block is actually a part of the current main chain of the
Tezos network. To avoid to be fooled by a fake chain, it is
<strong>necessary</strong> to carefully check that the block hash of the imported
block is included in the chain. This can be done by comparing the hash
to one provided by another node under the user’s control, or by
relying on social cues to obtain a hash from a large number of trusted
parties which are unlikely to be colluding.</p>
<p>As the Tezos position paper states:</p>
<blockquote>
<p><span class="dquo">“</span>Occasional checkpoints can be an effective way to prevent very long blockchain
reorganizations[…]. Forming a consensus over a single hash value over a period
of months is something that human institutions are perfectly capable of
safely accomplishing.
This hash can be published in major newspapers around the world, carved
on the tables of freshmen students, spray painted under bridges, included in
songs, impressed on fresh concrete, tattooed on pet ferrets… there are countless
ways to record occasional checkpoints in a way that makes forgery impossible.”</p>
</blockquote>
<p>This same wisdom must be applied when using a snapshot.</p>
<p>After that careful selection or verification of the imported block
hash, you can trust the node with the rest of the procedure. In
particular, you need not trust the source of the file, the snapshot
format contains everything necessary for the node to detect any
inconsistency, malicious or not.</p>
<p>This safety comes from the fact that block headers are designed to
make sure that applying a block has the same result for everyone in
the network. To achieve this, they include hashes of their operations
and predecessor, as well as the resulting chain state. The import
process makes the same checks, recomputing and checking all the hashes
it encounters in the snapshot.</p>
<h4 id="how-to"><a class="toclink" href="#how-to">How to</a></h4>
<p>To bootstrap a Tezos node from a file <code>FILE.full</code> (running this command
from an already synchronised node will not work), run:</p>
<div class="highlight"><pre><span></span><code><span class="err">$</span> <span class="n">tezos</span><span class="o">-</span><span class="n">node</span> <span class="n">snapshot</span> <span class="kn">import</span> <span class="nn">FILE.full</span>
</code></pre></div>
<p>Don’t forget to check the hash of the imported block displayed by the
node when importing.</p>
<h3 id="exporting-a-snapshot"><a class="toclink" href="#exporting-a-snapshot">Exporting a snapshot</a></h3>
<p>To export a snapshot, we first select a block hash which will
represent the point in history at which consumers of this snapshot
will start bootstrapping. By default, if no block hash is provided, we
automatically choose a block which was included in the chain a few
dozens of blocks ago. This is important as nodes bootstrapped from
this snapshot will not be able to reorganise their chain below this
block (they will set their checkpoint to this block).</p>
<p>Depending on the <code>snapshot export</code> option, additional history may also
be put in the snapshot file.</p>
<h4 id="how-to-create-a-snapshot"><a class="toclink" href="#how-to-create-a-snapshot">How to create a snapshot</a></h4>
<p>By default, the <code>snapshot export</code> command will create a <code>full</code>
snapshot. Such a snapshot will contain all the blocks from a given
block hash to the genesis. The whole chain will be exported into a
snapshot, from the beginning to the selected point. This kind of
snapshot can only be created from a <code>full</code> or <code>archive</code> node.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-node<span class="w"> </span>snapshot<span class="w"> </span><span class="nb">export</span><span class="w"> </span>--block<span class="w"> </span>BLOCK_HASH<span class="w"> </span>FILE.full
</code></pre></div>
<h4 id="how-to-create-a-rolling-snapshot"><a class="toclink" href="#how-to-create-a-rolling-snapshot">How to create a <code>rolling</code> snapshot</a></h4>
<p>This is the preferred use case if you want to deploy a node really
quickly or for test and experimentation purposes (such as a classroom)
as they are much smaller. However, to bootstrap a long running node on
the network, we recommend using <code>full</code> snapshots to participate into
the network wide preservation and sharing of chain history.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>tezos-node<span class="w"> </span>snapshot<span class="w"> </span><span class="nb">export</span><span class="w"> </span>--block<span class="w"> </span>BLOCK_HASH<span class="w"> </span>FILE.rolling<span class="w"> </span>--rolling
</code></pre></div>
<h2 id="on-garbage-collection-and-storage-optimisations"><a class="toclink" href="#on-garbage-collection-and-storage-optimisations">On Garbage Collection and storage optimisations</a></h2>
<p>The mechanism is in place for the node to run properly without the
archived context for a given block. But actually dropping these
contexts is another matter, usually known in the programming language
world as garbage collection (or <span class="caps">GC</span>). In garbage collection, when resources are no longer needed,
they are marked as such, and a garbage collector process comes from time to time
to clean them up and free the memory space they used.</p>
<p>This branch comes with an experimental <span class="caps">GC</span>, that can be run on startup
using the <code>--gc</code> option. Calling it on an archive node will do almost
nothing except cleaning up old discarded reorganisations. Calling it on
a rolling node after a cycle end should drop the archived contexts
before the newly set checkpoint. This <span class="caps">GC</span> is still slow, and cannot be
called while the node is running.</p>
<p>Several ongoing efforts (e.g. see plebeia, Irmin 2 and irontez) to provide an efficient
<span class="caps">GC</span> that can run transparently, and a better and more optimised storage
in general, are in progress. We shall discuss that matter more in a
later post.</p>
<h2 id="towards-other-history-modes-and-node-variants"><a class="toclink" href="#towards-other-history-modes-and-node-variants">Towards other history modes and node variants</a></h2>
<p>Nodes able to start and run with only a partial history open up many
variants and possibilities to adapt the disk and memory footprints to
different use cases.</p>
<p>Today, archive nodes have to start from the beginning, and validate the
entire chain. They don’t get synchronised until they have munched the
entire chain. We want to provide a version where nodes start in full
mode, but reconstruct their archives little by little from the genesis
while running. This way, we get both fast bootstrap from a snapshot,
and eventually archiving.</p>
<p>Another variant that we imagine is a node that keeps a sparse archive,
with an archive block at regular intervals. This way, it would be
possible to access intermediate archive contexts by validating only a
short sequence of blocks from the previous full block in memory. Much
like how compressed video formats work.</p>
<p>Finally, at some point we would like to release a node that requires even
less memory by not even keeping their full recent archives. Instead,
they would maintain and exponentially sparse history, and would have
to reconstruct their recent past when a reorganisation arises.</p>Nomadic Labs at Fosdem 20192019-02-01T16:00:00+01:002019-02-01T16:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2019-02-01:/nomadic-labs-at-fosdem-2019.html<p>Fosdem is of the largest gatherings of Free and Open Source Developers
in Europe. It’s held every year in Brussels during the first weekend of February.</p>
<p>This year Nomadic Labs is involved in the organization of the
“Blockchain and Crypto-currencies devroom”, a one day workshop to
present Blockchain projects to the <span class="caps">FOSS</span> community.
We strongly believe in open source and we want to share our work with
the <span class="caps">FOSS</span> community at large.</p>
<p>We will also present our work on the Tezos project, in particular how
on-chain governance could help the sustainable organization of online communities.</p>
<p>The program is available on the Fosdem
<a href="https://fosdem.org/2019/schedule/track/blockchain_and_crypto_currencies/">website</a>.</p>Meanwhile at Nomadic Labs #12019-01-21T13:00:00+01:002019-01-21T13:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2019-01-21:/meanwhile-at-nomadic-labs-1.html<p><img alt="Meanwhile..." src="images/meanwhile.png"></p>
<p>This post is the first of a series covering the on-going work at
Nomadic Labs.
In these posts, we focus on our current top priorities, but of course
we have many more projects in the works.</p>
<h2 id="preparing-our-first-amendment-proposals"><a class="toclink" href="#preparing-our-first-amendment-proposals">Preparing Our First Amendment Proposals</a></h2>
<p>Tezos is capable, by design, of automatically amending its economic
protocol without the need for a manual update or hard-fork.
The current protocol is designed to amend itself only after a
successful voting procedure, ensuring that there is awareness and
consensus in the community about the new features.</p>
<p>Nomadic Labs is currently preparing to submit its first protocol
amendment proposals for a vote.
Of course, it means selecting and implementing the contents of our
proposals, but also monitoring that the process behaves as intended.</p>
<p>Specifically, this includes making sure that the voting and test chain mechanisms
work as expected, and rehearsing the procedure.</p>
<p>It also means collaborating with our fellow Tezos core contributors to
make sure that everything is in place, including the new voting feature in
Obsidian Systems’ Ledger app.</p>
<p>More posts in the next weeks will prepare the ground for our proposal.
For the time being, bakers should familiarise themselves with the voting
procedure, fully described in <a href="https://medium.com/tezos/amending-tezos-b77949d97e1e">this
post</a>.</p>
<p>We hope to organise a rehearsal via the Alphanet in coming weeks.
We encourage bakers to join to test their setup is ready for the
potential upgrade.</p>
<h2 id="nomadic-labs-protocol-amendment-proposals"><a class="toclink" href="#nomadic-labs-protocol-amendment-proposals">Nomadic Labs’ Protocol Amendment Proposals</a></h2>
<p>We intend to propose two competing amendments. The first one increases the
current gas limit, the second one also increases the gas limit and, in
addition, it also reduces the roll size. This lets us, the community, test the
approval voting phase. Since one of these amendments may be the first to activate
on the Tezos network, we felt it important to propose simple changes.</p>
<p>The main goal is to prove that Tezos can carry out a successful update both
technically and as a community. After all, we are all novices to the Tezos
governance procedure and this aims to be a first step towards more substantial amendments.</p>
<p>Rolls, the unit of stake used to assign rights to bakers, are currently set at
10k ꜩ. Because roll size can significantly influence node performance, both in
terms of storage and computation time, we developed a specific set of
benchmarks to evaluate it. These benchmarks test the reassignment of the rolls
during the protocol upgrade as well as the evolution of the network for
several cycles afterwards. Changing the roll size is <em>not</em> just a matter of
changing a constant in one line of code!</p>
<p>Furthermore, roll size also has an economic impact on services that
build on Tezos, especially delegation.
Even if the community does agree to reduce roll size, it should be
carried out progressively to minimize disruption.
In a future post, we will describe further details about how rolls would be
reallocated during a potential upgrade.</p>
<p>The second important change that we want to propose is a (much
requested) increase of the gas limit for running Michelson smart contracts.
We thought it wise to set the gas limit as a conservative value
to protect the chain, with the intention that the Tezos amendment mechanism
facilitates future improvements based on a thorough benchmarking.</p>
<p>The current relative weight of Michelson instructions were established
on the basis of their algorithmic costs, using very conservative
factors to distinguish between time, space and I/O consumption.
Updating the limit to target an actual transaction throughput
requires the estimation to be more fine grained, using real-time
estimates on representative machines.</p>
<p>This performance analysis has been in the works for some time now and
it is starting to show practical results.</p>
<p>The proposed amendments leaves the current weighting as is, but does increase
the gas limit based on our preliminary results. In the future we hope to
completely refine the weight distribution of
instructions, and raise the limit once again based on the final benchmarks.</p>
<h2 id="storage-improvements"><a class="toclink" href="#storage-improvements">Storage Improvements</a></h2>
<p>Independently from the amendment proposal, we are also intending to release
an improved version of the node software which hosts the protocol.
The design of Tezos conveniently isolates the logic of the protocol inside
of a shell, which means changes that do not affect the protocol can be rolled
out more easily.</p>
<p>Several efforts towards a lighter disk footprint and faster
synchronisation of the node are almost ready to ship.</p>
<p>First, the node will be able to discard past states of the context using
a “garbage collector”. This means that no chain data is discarded
and no security property is lost in this process. Nodes will also, optionally,
be able to more aggressively discard older blocks to allow running on the more
size-constrained devices.</p>
<p>Second, the node will be able to bootstrap rapidly and safely using a
snapshot of a context that was saved previously or provided by another
node, without the need to re-download and replay all blocks since the
genesis. Of course, the safety of this mechanism depends on the user providing
a valid checkpoint on startup. This could be gathered from another node run
by the same user for instance.</p>
<h2 id="nomadic-labs-and-the-tezos-ecosystem"><a class="toclink" href="#nomadic-labs-and-the-tezos-ecosystem">Nomadic Labs and the Tezos Ecosystem</a></h2>
<p>We will also use this series of posts to highlight the few important
recent interactions we had with other members of the project.</p>
<p>Recently, a few of us have been visiting <span class="caps">NYC</span> to meet with other
members of the Tezos ecosystem, including <a href="https://obsidian.systems">Obsidian Systems</a>,
<a href="https://cryptonomic.tech/">Cryptonomic</a>, <a href="https://camelcase.com">CamlCase</a>, as well as Jacob and Alison from <a href="https://tqgroup.io"><span class="caps">TQ</span></a>.
We expect to collaborate more in the future, folks!</p>
<p>In October we organized a training session in Paris for new core
protocol developers, with some of the presentations available on our
<a href="https://www.youtube.com/channel/UCp9imUAFtU5kRyFKHN2iCDA">Youtube
channel</a>.</p>
<p>Nomadic Labs is also compiling new material for training sessions
targeted at smart contract and protocol developers that will take place in Q1.</p>
<h2 id="upcoming-in-depth-articles"><a class="toclink" href="#upcoming-in-depth-articles">Upcoming in-depth articles</a></h2>
<p>Some of the elements overviewed in this <em>Meanwhile at Nomadic</em> post will be
detailed in in-depth technical articles in the next few weeks.
Topics will include:</p>
<ul>
<li>Checkpoints, Storage Snapshots and <span class="caps">GC</span></li>
<li>How the Amendment Procedure Works</li>
<li>Benchmarking Michelson to Get Accurate Gas Accounting</li>
</ul>Study, Learn and Work with Nomadic Labs2019-01-19T13:00:00+01:002019-01-19T13:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2019-01-19:/study-learn-and-work-with-nomadic-labs.html<p>Nomadic Labs is opening its doors to students from all around the
world to join our internship program.</p>
<p>We are interested in many research and practical projects ranging from
<strong>parallel programming using hardware accelerators</strong>, to <strong>verification
of Michelson contracts</strong>, creation of <strong>ReasonML/Michelson decentralized
applications</strong> and <strong>privacy analysis of the Tezos network</strong>.</p>
<p>Internships are tailored to Master’s students. Some proposals have
already been submitted to a few European universities. They will
typically run for a maximum of six months.</p>
<p>We will happily receive and evaluate applications from students from
all around the globe. We will also be pleased to receive original
project propositions in related areas, and we welcome educators to
contact us to establish co-tutoring agreements.</p>
<p>We can also offer financial support to successful applicants to move
to France (Paris or Grenoble) for the duration of the internship.</p>
<p>Looking forward to <a href="mailto:contact@nomadic-labs.com">hearing</a> from motivated and enthusiastic students!</p>Nomadic Labs at POPL 20192019-01-15T09:40:00+01:002019-01-15T09:40:00+01:00Pietrotag:research-development.nomadic-labs.com,2019-01-15:/nomadic-labs-at-popl-2019.html<p><span class="caps">POPL</span> is one of the most prominent conferences in Programming Languages,
this year held in Portugal.</p>
<p>Programming Language Theory is at the core of the Tezos project as
demonstrated by the sponsoring of the Tezos Foundation for the conference.</p>
<p>Nomadic Labs is happy to attend the conference and looking forward to meeting
researchers and industrial partners interested in advancing the state
of the art in blockchain technology.</p>
<p>The program is available on the <span class="caps">POPL</span> <a href="https://popl19.sigplan.org/">website</a>.</p>Hello, world!2018-12-19T13:00:00+01:002018-12-19T13:00:00+01:00Nomadic Labstag:research-development.nomadic-labs.com,2018-12-19:/hello-world.html<p>We’re a team built around lead architects of the Tezos seed protocol,
Benjamin “klakplok”, Grégoire “hnrgrgr” and Pierre “chambart” of
OCamlPro with the help of the Tezos Foundation. After a busy year and
the successful launch of Tezos our team finally has a home: Nomadic Labs.</p>
<p>In the last year, our team has grown rapidly to more than 30 permanent
developers, coming from both academia and industry - many of whom hold
a PhD in computer science fields. Among them, Pierre Boutillier, our
release manager, is now well-known by the Tezos baking community.</p>
<p>After a few nomadic months, in November, we inaugurated our new office
in Paris. We look forward to hosting seminars, meetups, other
community events, and <a href="https://www.nomadic-labs.com/jobs.html">more developers!</a>
In the spirit of our name, Nomadic Labs extends beyond Paris. Indeed,
we also have a smaller office in Grenoble and a few colleagues are
based in other far-flung regions of the world.</p>
<p>Nomadic Labs houses a team focused on Research and Development. Our
core competencies are in programming language theory (and practice!),
distributed systems, and formal verification. We believe our strength
lies in a unique mix of skills and experience, allowing us to transfer
the best of academic research into real world applications.</p>
<p>We currently focus on contributing to the development of the Tezos
core software, including the smart-contract language, Michelson. We
also have a mobile team working on the Cortez wallet.</p>
<p>As a member of the Tezos community, we are fortunate to collaborate
daily with academic institutions and many other contributors to the
ecosystem. Some of our closest collaborators are researchers from the
French research institute
<a href="https://www.inria.fr/">Inria</a>,
<a href="http://ocamllabs.io/">OCaml Labs</a> in Cambridge (<span class="caps">UK</span>),
<a href="http://www.ocamlpro.com/">OCamlPro</a> and
<a href="https://tarides.com/">Tarides</a> in Paris,
<a href="https://obsidian.systems/">Obsidian Systems</a> and
<a href="https://cryptonomic.tech/">Cryptonomic</a> in <span class="caps">NYC</span>,
<a href="https://www.dailambda.jp/">Dailambda</a> in Kyoto
and, last but not least, many individual developers.</p>
<p>Stay tuned in 2019!</p>