Spectral joins the Rust Foundation

Choosing Rust to power Spectral’s core engine was firstly a matter of reason, but also, a lucky mistake. Coming from a painfully eclectic background of assembly, C++, .NET, Ruby, Go, Clojure, and Javascript, when I first tried Rust at around 2014, I saw the perfect balance, and pragmatic Zen in “all the best pieces combined”, but it was always breaking for me (this was before Rust was announced stable), so I gave up and went for Go.

But then, this happened

Years after that first time when I tried Rust, I found myself building the first iteration of the Spectral engine, it was prototyped in Go and C++. I just wasn’t sure about how Go is going to handle the performance and low-overhead requirements that I needed, so I added C++ along.

I needed something to power a developer-first security tool that can do everything: speed, safety, scale, stability, machine learning, low overhead, compactness, and extensibility. I quickly found out that Go wasn’t fast enough, and — after many hours chasing memory leaks, fiddling with all kinds of smart pointers as a remedy — I concluded that C++ was too hard to maintain.

Then, just for the heck of it, I went for Rust. I was porting my C++ code bit by bit, and slowly realized that just the porting experience felt amazing: the thinking model was already there, and I’m not actively busy thinking about pointers, safety, ownership, stack, heap, copying, the cost of this and that, and as the pieces were coming together, it also made a much more readable code base. The code looked like a pragmatically functional ML, friendly and accessible as Ruby, and as performant and powerful as C++.

But there’s more. When I finished, I did my first test run: the Rust implementation was faster, and consumed less memory than my C++ implementation.

What the hell?

To my awe, the Rust implementation provided performance, maintainability, and developer happiness by a very wide margin, and it ultimately won by simply carrying the momentum for a developer building a single solution for a painfully wide spectrum (- Spectral, get it?) of challenges: a security analyzer that can find all developer mistakes in code, data, configuration, and binaries that lead to security breaches, and have the same exact tool (from the very same codebase!) available for your IDE, command line, CI pipeline, containers, cloud hosts, log shippers and blob storage scanning.

Rust at Spectral

Now, for more than two years after that initial story, Spectral is a security startup, but Spectral the engine is also the core technology that does all these things that I mentioned before.

In a glimpse, Rust is delivering the following for us as a company, at scale:

• Head spinning performance
• Perfect maintainability
• A joy for building ML algorithms (explicit floating point accuracy and type safety FTW!)
• Number of bugs: virtually zero
• No redesigns
• Performance bottlenecks become very obvious and easy to optimize, never let down by the language
• If it compiles, it works

Also, here are some cool experiences we’re having with Rust:

• Using features massively for extra traceability and observability (special tracing builds), test-rig happy binary, and other use cases
• Swapping allocators for performance exploration
• Some ML algorithms were ported over from C/C++, Python, and we’ve gotten more accurate results as well as faster prediction
• Testing rig built with Node and Jest, for ease of test case generation, interoping with Rust perfectly at scale (running thousands of e2e test cases)
• Go embedding and interop (why not use both?)
• Heavy on the functional side, taking advantage of the ML heritage (strongly typed concurrency, data wrangling, and more)
• Enjoying the super fast, and popular parser ecosystem that Rust has
• Experimenting with compiling declarative security policies to WASM for speeding up execution times

Also, something that gets overlooked often when updating software – every Rust update does exactly what we expect from mission critical software: delivering more speed, and more stability, consistently. There were times when we updated Rust and our speed just went up, no code changes.

Looking ahead

Today we’re proud to say that Spectral is the ultimate developer-first security tool. From detecting secrets and sensitive data, to finding misconfiguration in any popular open source project (we aim at analyzing any project, not just Kubernetes!), to hardening CI/CD pipeline configurations and scanning massive amounts of data in S3 buckets with head spinning performance.

Rust and the Rust community has been a key player in realizing that for me personally and for us as a company. At Spectral, we feel obligated to give back as we move deeper into building security for developers at scale, based on machine learning (and some other tricks!).

Also, some necessary shoutouts:

• Rust foundation, Rust team, and workgroups – for obvious reasons. They’re the growth engine behind all this.
• Are we learning yet – for giving machine learning and Rust a stage. There’s a really powerful sweet spot for machine learning in Rust that I hope more people discover.
• The Rust Crypto team who’s been implementing cryptography for Rust – state of the art; and Tony Arcieri who I’ve been personally following since the old Ruby days.
• The Rust Analyzer team – which built rust-analyzer and saved us countless hours in productivity gained
• The Rust Reddit community, which I found warm, accepting and insightful