Notary Service at MongoDB

At MongoDB, we release new packages, binaries, and debug symbols with every commit that passes all the tests in our Continuous Integration system, and make them available for download. For some time we only provided MD5 checksums with our downloads. Packages that natively support signing - such as RPMs and Windows MSIs - were manually signed when creating a stable release. This meant that private keys had to be distributed to servers where users regularly logged in and manually interacted with them - increasing the risk that a key might be leaked or that the signing process may have problems. The manual process required for signing also meant that only stable release binaries were signed. Tarballs, which are the canonical way we distribute MongoDB, lacked signature files entirely.

We recently created and deployed an extensible tool to automate the generation of a variety of checksums and signatures for all our cross-platform packages and downloads with an automated “notary service.” The CI system submits artifacts to the notary service during the build process over a RESTful interface and gets all the checksums and signature files it needs returned. Keys for signing can be restricted to a single hardened server, with a single endpoint and API for signing tarballs, RPMs, and Windows MSIs. The distribution and hosting of checksums and signature files is also automated in the CI tool, reducing errors and ensuring that the signatures match the artifacts when they were produced, rather than as an after-effect of the release process.

In the talk, we would discuss the problems we had, the tool we created to solve them, and unexpected issues we ran into along the way to making our deployment of the tool scale in production.

Jonathan Reams is a build engineer at MongoDB, Inc. on the Core Server team. Currently he works on the toolchain and build system for the MongoDB server. Before joining the build team, he worked as a Systems Engineer on MongoDB’s DevOps team and Columbia University IT’s UNIX Systems Engineering group.