Update:On April 8, 2021, we released Stellar Core v16.0.0, which should fix the issue that caused several Stellar nodes to go offline earlier this week. All node operators should upgrade to that version as soon as possible to prepare for the full network upgrade scheduled for Saturday April 10, 2021 at 1500 UTC. If you have not done so by the time the network upgrades, your node will lose sync. To help with that, we published a Protocol 16 Upgrade Guide.
Early Tuesday morning, the Stellar Development Foundation’s validator nodes temporarily stopped processing ledgers, and the SDF public Horizon instance stopped ingesting them, which meant that for a brief period of time, it was unable to serve requests or submit transactions to the network. To resolve the situation, SDF’s engineering team worked quickly to spin up new nodes and a new Horizon instance, both of which were available by the end of the day.
However, several nodes run by others in the ecosystem were also affected, and so in parallel, SDF engineers continued to investigate the root cause of the problem. They identified it Tuesday evening. On Wednesday afternoon, they rolled out a patch release to address it, and on Thursday, they released Stellar Core v16.0.0, which should address any outstanding issues.
During the entirety of the SDF node downtime, there were still enough validators available to securely process transactions, and so the Stellar network remained online. Which is how a decentralized network is intended to work. Many validators also continued to publish archives that keep track of ledger history, and those archives will allow halted nodes to fill in gaps so they can recover quickly.
The following FAQ is intended to share what we know about the issue, and is a companion to a Protocol 16 Upgrade Guide that includes information to help affected organizations bring their ...