by Dirk McCormick (IPFS) and Edgar Lee (Netflix) on 2020-02-14
Web 2.0 services increasingly depend on large scale computing infrastructure to support fast iteration and deployment cycles. In order to build, test, and release software faster, these services are often seeking ways to upgrade their systems to take advantage of new learnings and tools from distributed peer-to-peer networks.
After IPFS Camp 2019, Netflix and IPFS began collaborating on ways to incorporate peer-to-peer services into Netflix’s developer tooling. Together, we figured out a way to leverage IPFS to speed up cloud builds, designing and testing solutions for faster Continuous Integration (CI) pipelines powered by efficient p2p container image distribution.
As part of this collaboration, we made big improvements over the last two quarters to Bitswap, the mechanism IPFS uses to transfer pieces of a file between two or more peers. A key factor enabling this improvement was p2plab - a performance benchmarking tool created by Netflix to run reproducible tests - allowing us to target and measure improvements. Our specific focus was a container distribution challenge Netflix wanted to address: how to efficiently pull container images in a large scale, multi-region environment. Image layers often reside in different regions. Leveraging IPFS as a peer-to-peer CDN lets nodes inside Netflix’s infrastructure collaborate and seed common pieces to neighboring nodes, helping make container distribution faster.
To speed this up even more, we added some useful new capabilities to the Bitswap protocol which cut transfer speeds in half for benchmarked use cases around container distribution. In the previous version of Bitswap, it took on average 9.08 seconds to pull a 300 MiB image to 32 leeching peers. The optimized branch brought this down to 3.16 seconds – 20% faster than Docker...