SE-Radio Episode 285: James Cowling on Dropbox’s Distributed Storage System

James Cowling of Dropbox (architect of their distributed storage system), speaks to Robert Blumen about their move from Amazon’s S3 to their own infrastructure; The show covers: the size, scope and scale of Dropbox’s data management; their experience on Amazon’s S3; why S3 over time did not meet their needs; how the decision was made to build their own; the architecture of Magic Pocket; building the skill sets they needed but did not have at the time; are layer violations ever a good thing?; his team’s “zero errors” policy; the important of invariants; how devops (sre) made the move easier; how they migrated; a brief excursion into the hardware side; what can engineers learn from this migration that will apply to other migrations (even smaller ones)?

Venue Dropbox headquarters, San Francisco

Related Links

• The Epic Story of Dropbox’s Exodus from the Amazon Cloud Empire: https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud-empire
• Inside the Magic Pocket (Dropbox blog post by James Cowling) https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pocket/
• Scaling to Exabytes and Beyond by Akhil Gupta: https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastructure/
• Pocket watch: Verifying Exabytes of Data by James Cowling: https://blogs.dropbox.com/tech/tag/magic-pocket/
• Infrastructure update: Pushing the edges of our global performance by Akhil Gupta: https://blogs.dropbox.com/tech/category/infrastructure/
• Guest Twitter: https://twitter.com/jamesacowling
• Guest email: contact Jane Rubin jane@thehatchagency.com with any questions

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | RSS