Ethereum phased synchronization: Refactoring the full synchronization mode (Full Sync)

Staged sync is refactored from Go-Ethereum's full sync mode for better performance.

Phased synchronization requires a lot of read and write operations. While our goal is to be able to sync nodes on HDDs, we still recommend using SSDs.

As the name implies, phased synchronization requires 10 phases to be executed sequentially.

The Turbo-Geth client learns from each peer the node's HEAD block (i.e. the latest block), and then executes each phase in turn, looking for the missing gap between the local HEAD block and the peer's HEAD block. blocks.

The first phase (downloading block headers) sets the local HEAD block.

The stages are executed sequentially. During the execution of each stage, the stage will not end until the local state of the node reaches the goal state.

That is, under ideal circumstances (no network outages, no application restarts, etc.), each stage only needs to be executed once to complete the initial sync.

After the last stage, the entire synchronization process will start again, looking for new block headers to download.

Paradigm researcher Samczsun released the Ethereum address tag and search-ethexc website: On September 6th, Paradigm researcher Samczsun released the Ethereum address Tag tag website tags.eth.samczsun.com on Twitter. Samczsun said, "I built a very simple website to label and label addresses, and anyone can contribute to it. You can search-ethexc by address, by label (with wildcards)."

0xScope, a knowledge graph protocol, said, “It’s great to see the attempt of open source address tags. If Samczsun and Paradigm are interested, we are also very happy to share our millions of address tag data.” [2022/9/6 13:12 :03]

If you restart the application between two stages, the application will restart from the first stage.

If you restart the app while a stage is executing, the app restarts from the current stage to complete that stage.

Through the pie chart below, we can see the time-consuming proportion of each stage (these are the data obtained from the full synchronization). Although these data are not precise, but enough as a reference.

Data: Ethereum 2.0 contract pledge added 4,032 ETH in the past 24 hours: According to the data of Okey Cloud Chain masters, as of 10:30 today, the Ethereum 2.0 deposit contract address has received 7.9386 million ETH, accounting for the current supply of Ethereum 6.77% of the volume, 4032 ETH was added in the past 24 hours. [2021/10/13 20:25:25]

If the blockchain reorganizes, we need to "roll back" some of the synced data.

Regression refers to regressing from the last stage to the first stage. However, one thing to note is that the transaction pool is not updated until after we perform the fallback, so we know the new nonce.

The phase order of the rollback is shown in the example below (happens sequentially from right to left).

state. unwindOrder = []*Stage{

   // Unwinding of tx pool (reinjecting transactions into the pool needs to happen after unwinding execution)

Coinbase: Ethereum and ERC-20 transaction delay issues resolved: According to official news, the Coinbase platform stated that the transaction delay issues of Ethereum and ERC-20 have been resolved. Officially, ETH and ERC-20 sending is fixed and should now work at normal speeds. [2021/2/13 19:40:29]

   stages, stages, stages, stages, stages, stages, stages, stages, stages, stages,

 }

Preprocessing via  ETL 

Some stages use our ETL framework to sort the data by key value before inserting it into the database.

This can greatly reduce the situation of database write amplification (write amplification).

So when we generate an index, or Hashed State, we go through a multi-step process.

News | Coinbase co-founder owns more Ethereum than Bitcoin: According to ccn reports, Fortune magazine recently released the latest list of the 40 most influential people under the age of 40, including Coinbase co-founder Brian Armstrong Ranked 20th. In its introduction, Fortune wrote that Armstrong now owns more ETH than BTC. [2018/7/21]

Write the processed data to several temporary files located in the data directory;

Then use a stack (heap) to insert the data in the temporary file into the database, and insert the data in the order that can minimize the database write amplification.

This optimization sometimes increases write speed by orders of magnitude.

Each stage contains two functions, which are ExecFunc  in the forward stage and  UnwindFunc in the backward stage.

In theory, some stages can work offline, but the current version does not implement this function.

Phase 1: Download block headers

At this stage, we download all block headers between the local HEAD block and the peer's HEAD block.

Bitfinex launches 12 new Ethereum token transactions: Bitfinex announced today that it will launch the following token transactions: Aion, IOSToken, Request Network (REQ), Raiden Network (RDN), Loopring (LRC), BnkToTheFuture Token (BFT ), Cofound.it, WAX, SingularityNET, MedicalChain, ODEM, and DAI. Deposit and withdrawal of all tokens has been enabled, and all tokens can be exchanged for BTC, ETH and USD, and the transaction will start at 24:00 on April 7th in Beijing. According to Bitfinex, the market value of these tokens exceeds US$1.1 billion, including decentralized trading protocols, payment solutions, blockchain 3.0 and other technologies, which are important asset supplements for Bitfinex. [2018/4/7]

This stage is CPU-intensive and is suitable for use with multi-core processors due to the proof-of-work required to verify block headers.

Most fallbacks start at this stage due to blockchain reorganizations.

This phase pushes the local HEAD pointer (pointing to the updated block).

Phase 2: Block Hash

Extract an index table that maps the block hash value to the block number (blockHash -> blockNumber) from the block header-ethexc to support faster lookup and make the synchronization process more friendly to mechanical hard disks.

Stage 3: Download block body

At this stage, we will also download the block body of the block header-ethexc downloaded in the previous stage.

A good internet connection is required for this phase. The vast majority of data is downloaded at this stage.

Phase 4: Restoring the sender

This stage recovers and stores the sender of each transaction in each downloaded block.

This stage is also CPU intensive and is suitable for use with multi-core processors.

No internet connection is required at this stage.

Stage 5: Executing blocks

At this stage, we execute every transaction in all previously downloaded blocks.

One thing to note is that during block execution we don't verify the root hash, or even create a Merkle tree.

This stage is single-threaded, does not require networking, and takes up a lot of disk space. If block execution fails, the stage can be rolled back.

Stage 6: Computing state roots

This phase builds the Merkle tree and verifies the root hash of the current state.

This phase also builds Intermediate Hashes and stores them in the database.

If no intermediate hashes were previously stored (which might happen during the first initial sync), this stage builds the full Merkle tree and its root hash.

If there are no intermediate hashes in the database, this phase uses the history of the block to figure out which hashes are outdated and which are the latest, and then use the latest hashes to build part of the Merk Er tree, only reconstructs stale hashes.

If the root hash fails to match, it goes back one block.

Phase 7: Generate hashed state

During execution, Turbo-Geth uses Plain state storage.

Plain State: In the standard state (which we call "hashed state"), the address of the account and storage item is  keccak256(address) , but in the general state, both The address of is address .

However, to ensure that some APIs work properly and remain compatible with other clients, we also generate hashed state.

If the hashed state is not null, we look at the History ChangeSet and only update items that have changed.

No internet connection is required at this stage.

Stage 8, 9, 10 : Index Build

3 indexes are generated during synchronization.

These 3 indexes may be disabled as they are not used by any API.

Transaction Query Index

The index table consists of a mapping from transaction hashes to block numbers.

Account History Index

The index stores a mapping from account addresses to a list of blocks in which the account's state has changed.

Store historical index

The index stores the mapping from the address of the storage item to the list of blocks where the storage item has changed to some extent.

At this stage, we start the transaction pool or update its state. For example, if some transactions are included in the blocks we have downloaded, these transactions are removed from the transaction pool.

When rolling back, we will re-add the transactions in the rolled back block to the transaction pool.

Original link:

https://github.com/ledgerwatch/turbo-geth/tree/master/eth/stagedsync

By Alex Sharov

Translation & proofreading: Min Min & A Jian

Tags：

Blockchain