We need to protect the use of secrets in the vdc. There is plethora of secrets in the default namespace. There are several in batch-pods. There is one in test.
We also need to protect the use of k8s service accounts in the vdc. There are several in the default namespace. There are three in batch-pods. There is one in test.
We also need a system to generate secrets from some root credentials (e.g. a hail team member’s broad login credentials).
Currently batch allows anyone to mount any secret and use any service account in the batch-pods namespace (or test namespace, for PRs that are testing batch or ci). Batch runs in two contexts:
- main batch instance in default (creating pods in batch-pods)
- PR batch instance in batch-pods (creating pods in test)
Batch clients come (or will come) from five places:
- ci in default
- pipeline from a notebook (so, protected by an authentication check)
- PR batch-tests in batch-pods
- PR ci in batch-pods
- PR pipelines in batch-pods
We must prevent a PR from leaking a secret in logs or by sending it to the public internet. We currently ignore malicious notebook pipeline users (we assume Konrad and Liam are careful and good actors).
A couple concrete proposals:
- pods in
batch-podsshould not be permitted to talk to the
batch(ci & pipeline already spin up a batch instance which creates pods in
- CI should only test PRs created by whitelisted users (these people: https://github.com/orgs/hail-is/people)
- If a hail team member approves a PR, then a PR created by a non-whitelisted user will be tested
- Hail team reviewers must verify a third-party PR does not insecurely handle secrets.
I think when we have untrusted notebook pipeline users, we will need batch to authorize access to service accounts and secrets.
Regenerating secrets is also important, but I haven’t put much thought into that yet.