Dropbox Open Sources Go Libraries Following Migration From Python Code Base – InApps 2022

Dropbox, like a lot of companies, has started to adopt the Go programming language.  Yesterday, the company announced it was open sourcing its libraries to help the broader community build large scale production systems.

About a year ago, Dropbox started migrating its performance critical systems off Python. The company adopted Go to scale its systems in a more efficient manner. Since then, the company has moved major parts of its infrastructure to Go. But in the process, it faced a problem with Go that is a reflection of the programming language’s youth more than anything else.

Go did not have the depth of libraries that Dropbox needed to build larger systems.  To address the issue, the Dropbox team started to build its own libraries that offer better abstractions, such as connection management and a memcache client. 

Here’s what it is offering to kickstart the effort as described by Patrick Lee:

• caching: provides a general abstraction for building caching layers.
• errors: enhances the standard error interface and exposes stack trace information.
• database/sqlbuilder: allows developers to generate sql statements programmatically.
• memcache: implements a fully featured memcache client library which supports connection pooling and flexible sharding.
• net2: adds functionality for connection management.
• (and my personal favorite, hash2, which includes a space efficient permutation-based consistent hash algorithm)

Dropbox says it will continue to expand the libraries it uses and offer them publicly on GitHub. These are the same libraries Dropbox will use internally.