announcements

Stopping Supply Chain Attacks With Preflight

By Dotan Nahum May 31, 2021

It’s been a few weeks since the CodeCov hack sent ripples across the software development industry. One of these ripples was an increased awareness of supply chain attacks and the risks they present. That said, it’s easy to see that this awareness is not yet implemented as a best practice in the field.

Here are a few examples:

These are all different forms of remotely running shell scripts and blindly executing them. Believe it or not, these examples are live right now in each of these projects, weeks after the CodeCov breach was discovered and mitigated. An attack in which, as you may remember, this very script was altered to contain malicious code.

But the CodeCov breach is only a symptom of a bigger, more complex issue – supply chain attacks.

CodeCov and Supply Chain Attacks

The above examples of remote CodeCov bash execution may vanish over time. However, other instances of remote shell script execution are prevalent even across projects and repositories on GitHub.

These create an attack vector for attackers and malefactors. There are numerous types of attacks in this space, with different motivations behind them.

Malicious scripts
Since an install script, or a “quick tool” script, is by itself runnable software, the script can be replaced with any other piece of code that performs an attacker’s goal
Swapping software
A script that we pull can still feel like it behaves nicely. But then, it would let us install a different software package than what we originally thought it would.
Tricking into old updates
To avoid being detected for giving you a tainted binary, an attacker may simply push an older version of the same software you’re installing, assuming that this older version contains a famous exploit they can use.
Tricking into future updates or “update jacking”
An attacker might want to point the script at their own update package. Which, in terms of server, is well into the future. So rollback attempts would fail. For the same goal, an attacker can replace a specific update in the past, present, or future (as long as they can cause an updater to trigger).

At the time of writing, these were all live:

Granted, we haven’t discovered anything new here — just a best practice still waiting to become a best practice. But what would that best practice be?

According to CodeCov and their own analysis of their breach:

The answer seems quite simple – validate and verify remotely executed bash scripts.

Introducing: Preflight

We’ve built Preflight as a tool to implement that advice into a best practice that is easy to use and integrate into your DevOps workflow. With a few more features and the icing on top, preflight is now a platform for checking, validating, and running 3rd party components aimed to help you block supply chain attacks.

preflight helps you verify scripts and executables and mitigate supply chain attacks.

How Does Preflight Work?

What happens when you execute a “curl | sh” karate-chop? Here’s one breakdown of such a chain in very general terms, highlighting the major links in the chain:

When we pull a shell script from a website, we’re trusting the tool we use (curl), and the OS for SSL (CA chain), and the domain (and by derivative our DNS services), and the human operating the domain, and the code hosted on the domain, and the human operating the hosting and code. That’s quite a few links in a chain.

With Preflight we’re moving the focus of the chain of trust to a single place: the end of the chain. A signature of the executable which you want to run. The signature is produced and maintained by you, and only you.

Although there are many ways to verify trust throughout the entire chain, these would be of more value to software producers (think software factory) to enable forensics, traceability, and more.

For our use case, and as depicted in the original Codecov hack, we can get all the benefits of securing our usage of the 3rd party software as consumers. Which is simpler than considering the entire chain of supply on the producer side.

Getting Preflight

First of all, it’s the chicken and the egg. How do you pull a legit preflight binary from us without verifying it with preflight, having that preflight solves this exact problem?

The best way is to grab the source, compile it yourself, and use your own binary which you put in a place that you trust. You will usually have several options to do that safely:

Put it on your own S3 bucket
Drop it on your own Artifactory or similar
Push it directly into your repos (it should be as small as 4mb and seldom change so Git should work nicely with it)

If you want to just get started quickly on your workstation, you can download a release or install preflight with homebrew:

$ brew tap spectralops/tap && brew install preflight

Flying With Preflight

The curl | sh workflow is packaged as a tool, verified with our own signature that we can create with Preflight. In addition, Preflight can verify any runnable against several of providers such as VirusTotal and others to detect malware automatically

So how does it look in action?

Someone changed the script or binary you’re running. Abort!

$ curl -L https://XXX | preflight run sha256=1ce...2244a6e86
⌛️ Preflight starting
❌ Preflight failed:
Digest does not match.

Expected:
<...>

Actual: 
<...>
  
   Information:
   It is recommended to inspect the modified file contents.

A hash is verified, but it is actually vulnerable. Abort!

$ curl -L https://XXX | preflight run sha256=1ce...2244a6e86
⌛️ Preflight starting using file lookup: malshare.current.sha256.txt
❌ Preflight failed: Digest matches but marked as vulnerable.
   Digest matches but marked as vulnerable.

Information:
  Vulnerability: Hash was found in a vulnerable digest list
  More: malshare.current.sha256.txt

All ok, let’s fly.

$ curl -L https://XXX | preflight run sha256=1ce...2244a6e86
⌛️ Preflight starting
✅ Preflight verified

... actual script output ...

Codecov Revisited

A perfect example of Preflight in action would be CodeCov.

First, let’s create a hash. Before creating it, be sure to review the script manually and see that it’s not doing anything funny:

$ curl -s https://codecov.io/bash | ./preflight create
sha256=d6aa3207c4908d123bd8af62ec0538e3f2b9f257c3de62fad4e29cd3b59b41d9

Now, we’re going to take

sha256=d6aa3207c4908d123bd8af62ec0538e3f2b9f257c3de62fad4e29cd3b59b41d9

And use this to secure our pulls from CodeCov. In this case, preflight is checked safely into your repo under ci/preflight.

BEFORE (insecure):

steps:
   - curl -s https://codecov.io/bash | sh

AFTER (safe, yay!):

steps:
   - curl -s https://codecov.io/bash | ./ci/preflight run sha256=d6aa3207c4908d123bd8af62ec0538e3f2b9f257c3de62fad4e29cd3b59b41d9

Dealing With Change

When updating an old binary or script to a new updated version, there will be at least two (2) valid digests “live”. Just replacing the single digest used will fail for the older runnable which may still be running somewhere.

$ preflight <hash list|https://url/to/hash-list>

To support updates and rolling/auto-updates of scripts and binaries we need to validate against <old hash> + <new hash> at all times, until everyone upgrades to the new script. Preflight validates against a list of hashes. Or better: give it a live URL of valid hashes and it will validate against it.

curl .. | ./ci/preflight run sha256=d6aa3207c4908d123bd8af62ec0538e3f2b9f257c3de62fad4e29cd3b59b41d9,sha256=<new hash>,...

Or to a live URL:

curl .. | ./ci/preflight run https://dl.example.com/hashes.txt

Use this when:

You employ multiple digests verbatim and when your runnables change often (but not too often)
You use a URL when your runnables change often. Remember to follow the chain of trust. This will now mean that:
- Your hash list URL is now a source of trust
- Visually: we’re swapping the chain of trust like so curl <foreign trust> | ./ci/preflight <own trust>

Checking Scripts And Binaries

If you want to use Preflight as another tool for improving your Unix pipes chops, you can with the “check” command:

Piping:

$ curl -s https://example.com/some-script | preflight check sha256=d6aa3207c4<...>b4 | sh

Note that preflight check is built in a way that allows you to continue to pipe to the next process:

If a check passes, the checked script or binary content will be dumped to STDOUT
If a check fails, you’ll get an exit(1), and an error message

Executables:

$ preflight check sha256=d6aa3207c4<...>b4 ./my-script.sh

In this case:

If a check passes, you’ll get an exit(0) and no output (so you can compose with other tooling)
If a check fails, you’ll get an exit(1) and an error message

Creating New Hashes

You can easily create new hashes with preflight. The default is a SHA256 hash, but you could also create a sha256, sha1, and md5 hash.

$ preflight create test.sh
sha256=fe6d02cf15642ff8d5f61cad6d636a62fd46a5e5a49c06733fece838f5fa9d85

Though not recommended, you can create other kinds (weaker kinds) of hashes for legacy/compatibility reasons:

$ preflight create test.sh --digest md5
md5=cb62874fea06458b2b0cabf2322c9d55

Malware lookup

preflight comes with lookup providers. This is optional and you can enable them by using environment variables:

File Lookup

You can download a daily list of malware signatures from malshare.com or any equivalent service. Here is a direct link to such a list.

Then:

Set PF_FILE_LOOKUP=./path/to/text/file

With this configured preflight will search for all digest types in this file before approving.

Here is a full example for your CI, combining preflight with Malshare:

env:
   PF_FILE_LOOKUP: malshare.current.sha256.txt

steps: 
- wget https://www.malshare.com/daily/malshare.current.sha256.txt
- curl https://... | preflight <sha>

Result:

$ PF_FILE_LOOKUP=malshare.current.sha256.txt preflight run fe6d02cf15642ff8d5f61cad6d636a62fd46a5e5a49c06733fece838f5fa9d85 test.sh
⌛️ Preflight starting using file lookup: malshare.current.sha256.txt
❌ Preflight failed: Digest matches but marked as vulnerable.

Information:
  Vulnerability: Hash was found in a vulnerable digest list
  More: malshare.current.sha256.txt

VirusTotal Lookup

You can use the virus total community API access to lookup your hashes.

Set PF_VT_TOKEN=your-virustotal-api-key

With this configured, preflight will automatically create the VirusTotal lookup provider and validate the digest with it.

Here is a full example for your CI, combining preflight with VirusTotal:

env:
   PF_VT_TOKEN: {{secrets.PF_VT_TOKEN}}

steps: 
- curl https://... | preflight <sha>