Skip to content

Worker Secrets

A worker that needs credentials, such as an API key for an exchange, gets them by reference. CreateWorkerRequest.secrets maps an attribute name to a key of the deployment's worker secret:

{
  "mime_type": "application/x-dotnet-dll",
  "topics": ["x.myapp.handshake"],
  "config":  { "symbols": "BTC/USD,ETH/USD" },
  "secrets": { "alpacakey": "alpaca-paper-key", "alpacasecret": "alpaca-paper-secret" }
}

The worker reads the value like any config entry, as an extension attribute of the triggering CloudEvent: input["alpacakey"].

Why not config or tags?

Config and tags are fine for settings, but not for credentials:

  • ListWorkers returns them to every API-key holder.
  • The WorkManager stores them in plain text in its state store.
  • Tags can also be changed after creation, and filtered on.

With secrets, only the references (alpaca-paper-key) are stored and listed.

How it works

  1. CreateWorker validates the references and stores them with the worker. Values never reach the WorkManager.
  2. When the worker loads, the WorkManager Runtime reads the worker secret from its Dapr secret store and resolves every reference. If one is missing, the worker becomes LOAD_FAILED, and the load_failed event names the missing reference.
  3. On every trigger the Runtime stamps the values onto the CloudEvent, after the event's own attributes: a publisher cannot substitute a worker's credentials.
  4. The Runtime drops the secret attributes from everything the worker returns, so a worker that hands its input back never publishes them.

Values are held in memory only, and refreshed every WORKER_SECRETS_TTL_SECONDS (60 by default), so a rotated secret reaches running workers within a minute. If the store is unreachable, the last values are kept.

Rules

What Rule
Name A CloudEvent extension attribute name: 1-20 lower-case letters or digits
Name Not a reserved attribute (CloudEvents core attributes, correlationid, replytopic, publishtopic, traceparent, tracestate)
Name Not also a config key
Reference [A-Za-z0-9._-]{1,253}, the Kubernetes Secret key rules
Worker type Runtime-hosted types only (text/x-csharp, application/x-dotnet-dll, application/x-native-dll, text/x-python, application/x-python)

A request that breaks a rule fails with INVALID_ARGUMENT. Secrets, like config, cannot be changed after creation: recreate the worker.

Deployment

The Runtime reads one secret from one Dapr secret store:

Environment variable Default Meaning
WORKER_SECRETS_STORE workersecrets Dapr secret store component
WORKER_SECRETS_NAME virtufin-worker-secrets The secret whose entries are the keys
WORKER_SECRETS_TTL_SECONDS 60 How long fetched values are reused

On Kubernetes, the Helm chart (workerSecrets values) defines the store as secretstores.kubernetes, scoped to the Runtime. It runs the Runtime under its own service account, which may read exactly that one Secret. So no worker can reference the API's own keys or the registry token. The Secret itself is created outside the chart:

kubectl -n virtufin create secret generic virtufin-worker-secrets \
  --from-literal=alpaca-paper-key="$ALPACA_API_KEY" \
  --from-literal=alpaca-paper-secret="$ALPACA_API_SECRET"

deploy-local runs no WorkManager Runtime, so worker secrets are not available there.

Managing the Secret

On medusa the Secret is managed by Ansible (k3s-common role virtufin_backend): it is built from VIRTUFIN_WORKER_SECRETS, a JSON object key -> value in the cluster .env, on every deploy. Change it there and redeploy; edits made with kubectl are overwritten by the next deploy. Elsewhere, use kubectl as below. Keep values out of your shell history: read them from a file or from variables, never type them on the command line.

Task Command
Create the Secret, or replace all its keys kubectl -n virtufin create secret generic virtufin-worker-secrets --from-env-file=worker-secrets.env --dry-run=client -o yaml \| kubectl apply -f -
Add or change one key kubectl -n virtufin patch secret virtufin-worker-secrets --type merge --patch-file=patch.json with {"stringData": {"my-key": "..."}} in patch.json
Remove one key kubectl -n virtufin patch secret virtufin-worker-secrets --type json -p '[{"op":"remove","path":"/data/my-key"}]'
List the keys (never the values) kubectl -n virtufin get secret virtufin-worker-secrets -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}'

worker-secrets.env holds one key=value per line and stays out of git. Keys follow the reference rule above ([A-Za-z0-9._-]).

What a change does to running workers:

  • Changed value: every worker that references the key gets the new value within WORKER_SECRETS_TTL_SECONDS (60 s). No worker has to be recreated.
  • New key: only workers created afterwards can reference it. A worker's references are fixed when it is created, like its config.
  • Removed key: within the refresh interval, every trigger of a worker that references the key fails with secret '<key>' not found. The next time that worker loads it goes LOAD_FAILED. Add the key back, or delete the worker.
  • Store unreachable: the Runtime keeps the last values it fetched and logs a warning, so running workers continue. A worker that loads for the first time on a pod without cached values fails to load.

Any other Dapr secret store works too, such as Vault or a cloud key vault: point WORKER_SECRETS_STORE at it, and make WORKER_SECRETS_NAME a secret that holds a map of keys.

Limits

  • Every API-key holder can create a worker that references any key of the worker secret, and that worker can use or leak the value. Prefer low-privilege or expiring credentials.
  • A WebSocketManager handshake handler sends the login it builds over pub/sub to the WebSocketManager (see the WebSocketManager's handshake docs, "Security"). Worker secrets don't change that.