Choosing a programming language for a new project is a rigid architectural decision; after selecting a language, there is no changing it without a complete rewrite. This article discusses why we chose Rust as the programming language for one of our critical software components, what has gone well, and some of the challenges we’ve encountered.
Rust at Aembit
Aembit is a platform that assigns identities to workloads and controls access to the services they interact with. The Aembit platform consists of two components:
- A cloud-based management plane that includes the admin console and the system backend
- A proxy that sits in line with customer traffic and performs actions as directed by the management plane
This article focuses on our language selection for the proxy component.
Our Proxy-Language Shootout
The obvious choice for our team was C++. It is a language the engineers on the team used when building previous products, and the developers of most modern proxies use it too. However, C++’s memory management and concurrency dangers had bitten these folks in the past, and it wasn’t easy to feel confident in the safety of the code, especially as new members joined the team with varying coding ability and maturity levels.
The next contender was Go, which our team had some experience with. Go is a common choice for network programming due to its high performance, concurrency support, and safety. Since Go is a garbage-collected language, it would protect against many of the memory issues that can plague C++ code. However, the garbage collector presents another type of challenge for high-performance applications. While Go’s garbage collector has a good reputation overall, high-performance applications may exhibit problems where garbage-collection pause impedes performance. Even though Go offers a few ways to tune the garbage collector, there isn’t a way to circumvent all potential challenges. While we are generally cautious about over-optimizing for performance before having a good understanding of performance bottlenecks, the possibility of needing to rewrite the proxy in a more performant language later was too significant of a risk for us as a fledgling product.
Our third option, and our ultimate pick, was Rust. Rust combined the performance of C++ with the memory safety guarantees of Go. However, there was a minor challenge in that no one on our team had experience with Rust. Our first engineer began by reading Programming Rust, which was sufficient to build out an initial prototype of the proxy after about a month and a half of experience. Note that this engineer has a lot of past development experience and is not something a less experienced person likely would have been able to do.
What Went Well
We wouldn’t have been able to make as much headway, especially given our team’s minimal Rust experience, without the help of Rust communities sprinkled all over the internet. Stack Overflow, The Rust Programming Language Forum, and project-specific communities across Discord, Slack, and Gitter were instrumental in getting us unstuck during the prototyping phase and beyond.
Rust’s tooling ecosystem is well developed. Rust’s package manager and general swiss-army knife, called Cargo, makes building, testing, running, and dependency management trivial. It also has enabled our CI process easily via GitHub Actions. Other tools that we’ve leveraged with success include:
- Formatting (via Rustfmt)
- Linting (via Clippy)
- Language server (via rust-analyzer) and the associated VS Code plugin
- Debugging (via LLDB)
- Native unit and integration testing
- Code coverage (via LLVM)
So far, developing in Rust has given us a slight confidence edge that we may not otherwise have had. Rust’s Result and Option types make error handling explicit and exhaustive, and Rust reduces runtime errors since it doesn’t have the concept of null. We can find almost all places where runtime errors can occur by searching for “unwrap” and “expect” methods in the codebase.
Rust performance is on the same order of magnitude as C and C++. Since it lacks garbage collection, we have no concerns about our ability to tune our application to whatever resource and performance requirements our customers may demand in the future. We are confident in optimizing our code to the degree it is needed.
Lastly, Rust’s memory safety guarantees eliminate a class of errors we probably would have had if we had gone with C++. That has given us incredible peace of mind and freed up mental bandwidth to focus on other priorities instead of catching memory vulnerabilities.
Rust is a big language with some challenging concepts, such as ownership and lifetimes. As one of our engineers pointed out, learning Rust concepts is often challenging due to the layered complexity. One can’t just dive into a topic such as asynchronous code without an in-depth understanding of the memory model, lifetimes, and other topics. That makes it tricky for someone starting with Rust to jump into the deep end without thoroughly understanding what is happening. The same engineer noted that a beginner in C++ can produce code sooner than a beginner in Rust. Still, the C++ code they write when first starting will be dangerous, whereas Rust’s compiler will prevent unsafe code from compiling in the first place.
Hiring Rust programmers is tricky. Since it is still a relatively new language, it is hard to find people who are well-versed in the details of the language. It took us four months to find a senior developer with Rust experience who was the right fit for where we are as a team.
Prototype to Production
In our experience, heavily rearchitecting a design in a Rust codebase can be agonizing. Rust’s strict rules to support safety without garbage collection force you to adhere to Rust-friendly architecture. As a team, we have not yet developed the experience with Rust to foresee what is cleanly implementable in Rust and what isn’t. As a result, we hit multiple roadblocks when trying to migrate from our initial prototype to our second draft. It is a common saying that Rust forces you to think about your designs upfront. That is a challenge for quick iteration.
If we were starting over again, would we have used Rust again? We’re still not sure. While we are glad we picked Rust instead of C++ for our memory peace of mind, we still wonder whether Go may have been sufficient for our purposes. Go seems to address all of the challenges listed:
- It has a much simpler learning curve.
- It is easier to find programmers that have deep experience with it.
- Its garbage collection means dramatically easier rearchitecting.
However, with Rust, we are not losing any sleep about our ability to optimize performance and feel more confident than we would otherwise in our runtime stability. If you are in a similar situation where you are evaluating Rust and Go, we recommend carefully considering the challenges we’ve highlighted above. Consider carefully whether the speed and runtime stability afforded by Rust justify the practical hurdles.
Aembit is the Identity Platform that lets DevOps and Security manage, enforce, and audit access between federated workloads. To learn more or schedule a demo, visit our website.