Work with secret files

We want to work as much as possible in the open, but we still require some secrets to deploy hubs securely. There are two primary kinds of secrets:

  1. Parts of JupyterHub config (such as proxy.secretToken or OAuth2 secrets) that need to be passed to helm. These are stored as .yaml files under deployments/<name>/secrets.
  2. Credentials for authenticating to cloud providers for automatic deployment - such as GCP Service Account keys. These are a little more sensitive, since they give attackers access to our infrastructure.

We protect all these with sops, a wonderful open source project maintained by Mozilla. We use it to do encryption / decryption via a cloud managed key management service (in our case, GCP KMS). This allows us to do access control via Google Cloud permissions, and makes sure the secrets are encrypted at rest.

Setting up sops

To view / edit the secrets, you need:

  1. Access to the appropriate cloud KMS key. You can find out what key this is by looking at the .sops.yaml file. Currently, it is the key jupyterhub-deploy in the keyring sops-keys in the project ``two-eye-two-see`. Someone with access to the project needs to grant you the appropriate rights.
  2. The sops command installed.
  3. Authenticate to Google Cloud with the gcloud commandline utility.

Creating / Editing secrets

Once you have these, you can view / edit the files easily.

Run sops <filename> with the name of the file you want to view / edit. This will open the file in the editor mentioned by your $EDITOR env variable (defaults to vim). You can edit the file here, after which you can save & exit. sops will re-encrypt your file after this, so the unencrypted form is available for as little time as possible.

You can create a new encrypted file in exactly the same way. The .sops.yaml file tells sops which keys to use for any file created under a directory named secrets. So we should put all our secrets under directories named secrets (or their subdirectories).

Warning

You must create new files with the sops command. If you create them in other ways, they will not be encrypted! This is very bad, since our repositories are all public.

Make sure to check the output of git show after you make a commit with a new secret file before pushing.