Sepolia Incident

The Sepolia Incident: A Threat to the Ethereum Network

On February 6, 2024, a critical vulnerability was discovered in the Ethereum network, specifically in the Execution and Consensus Layer clients. The issue, known as the Sepolia incident, posed a significant threat to the network's security and stability. In this article, we will delve into the details of the incident, its impact, and the remediation efforts that followed.

Background: The Merge and RPC Size Limits

Prior to the Merge, different message size limits were set for RPC communication to protect clients from denial-of-service (DOS) attacks. These limits were carried over to the engine API, which plays a crucial role in connecting Execution and Consensus Layer clients during block production. However, due to the engine API's involvement in block production, it became possible for blocks to be produced that surpassed the RPC size limits of some clients but remained within the acceptable range for others.

The Vulnerability: Crafting a Malicious Block

An attacker could create a message that exceeds the size limit of the client with the lowest setting, while still adhering to the gas limit requirements. By waiting for a block to be produced, the attacker could create a situation where some clients regard the block as valid, while others reject it, issuing a HTTP error code "413: Content Too Large." This would allow the attacker to force the majority of nodes to reject blocks that a minority would accept, resulting in a forked chain and lost rewards for the proposer.

The Initial Attack: Builders and Modified Clients

Initially, it was thought that only builders or a modified version of a client could create these malicious blocks. However, it was later discovered that a client with a higher limit could propose a block, and the Consensus Layer would request validation of this proposed bigger block. This made it possible for an attacker to create a malicious block without needing a modified client.

The Bigger Issue: Transactions Below 128KB

On February 7, 2024, it was discovered that an attacker could create a block that would hit the 5MB limit with a bunch of transactions that are below the 128KB limit and not exceed 30 million gas. This was a bigger issue because an attacker could create a bunch of high-paying transactions and send them to the network. Since they outpay everyone else in the mempool, every node (even geth nodes) would include the attack transactions in their block, creating a block that would not be accepted by the majority of the network.

Timeline of Events

Here is a timeline of the events surrounding the Sepolia incident:

February 6, 2024, 13:00: Toni, Pari, and Justin try to submit a specific transaction to the network.
February 6, 2024, 13:25: Pari receives errors from his local Geth node.
February 6, 2024, 15:14: Justin manages to put the transaction in a block and submits it through the Besu client.
February 6, 2024, 20:46: Sam alerts Pari, Toni, and Alex about certain Sepolia nodes struggling.
February 6, 2024, 21:05: The team double-checks with Marius from Geth and confirms the bug.
February 6, 2024, 21:10: The team gets together to debug the issue.
February 7, 2024, 23:40: The team decides to limit the RPC request limit to 5MB.
February 7, 2024, 6:40: The team discovers that there might be a bigger issue and the attack can be executed with transactions less than 128KB size.
February 7, 2024, 10:00: The team decides to increase the RPC request limit.
February 7, 2024, 21:00: The fix is merged in Geth.
February 9, 2024: Geth is released.

Remediation Efforts

While Geth was the only client affected by this bug, other clients have also updated their defaults to be safe from this attack even if gas limits are increased. The client teams have indicated that the following updates have the safe RPC limits:

Geth: v1.13.12
Nethermind: v1.25.4
Besu: 24.1.2
Erigon: v2.58.0
Reth: v0.1.0-alpha.18

Conclusion

The Sepolia incident highlights the importance of security and stability in the Ethereum network. The vulnerability that was discovered posed a significant threat to the network's security and stability, but the remediation efforts that followed were swift and effective. The incident serves as a reminder of the importance of ongoing security and maintenance efforts to ensure the continued stability and security of the Ethereum network.

Forward-Looking Thoughts

The Sepolia incident has important implications for the future of the Ethereum network. As the network continues to evolve and grow, it is essential that security and stability remain top priorities. The incident highlights the need for ongoing security and maintenance efforts to ensure the continued stability and security of the network. Additionally, the incident serves as a reminder of the importance of collaboration and communication among the Ethereum community, including developers, researchers, and users. By working together, the Ethereum community can ensure the continued security and stability of the network, and ultimately, the success of the Ethereum ecosystem.

Source: https://blog.ethereum.org/en/2024/03/21/sepolia-incident