Skip to content

Worker Lifecycle

Workers go through several state transitions during their lifetime. This guide covers creation, state management, and cleanup.

Worker States

stateDiagram-v2
    [*] --> Stopped : CreateWorkerAsync()
    Stopped --> Running : StartWorkerAsync()
    Running --> Stopped : StopWorkerAsync()
    Stopped --> Running : StartWorkerAsync()
    Running --> Deleted : DeleteWorkerAsync()
    Stopped --> Deleted : DeleteWorkerAsync()

Creation

CreateWorkerAsync

var workerId = await workManager.CreateWorkerAsync(new CreateWorkerRequest(
    CodeSource: new CodeSourceContent(Encoding.UTF8.GetBytes(pythonCode)),
    MimeType: "text/x-python",
    Topic: "my-topic"
));

What happens:

  1. Engine lookup: Validates that an engine exists for the MIME type
  2. Code fetch: Extracts code from URL or uses inline content
  3. Code loading: Calls engine.LoadCodeAsync(code)
  4. Persistence: Saves worker config to Dapr state store
  5. Subscription: Creates Dapr pub/sub subscription
  6. Registry: Adds worker to in-memory registry
  7. Status: Sets worker to Stopped (requires explicit StartWorkerAsync to begin processing)

CodeSource Types

Inline Content: 

CodeSource: new CodeSourceContent(Encoding.UTF8.GetBytes(code))

From URL: 

CodeSource: new CodeSourceUrl(new Uri("https://example.com/worker.py"))

State Transitions

StartWorkerAsync

Resumes a stopped worker:

await workManager.StartWorkerAsync(workerId);

What happens: 1. Re-establishes Dapr topic subscription 2. Sets status to Running

Notes: - Idempotent: If already running, does nothing - Worker continues processing from where it left off

StopWorkerAsync

Pauses a running worker:

await workManager.StopWorkerAsync(workerId);

What happens: 1. Removes Dapr topic subscription 2. Sets status to Stopped

Notes: - Idempotent: If already stopped, does nothing - Worker can be restarted with StartWorkerAsync

Code Updates

LoadCode

Updates worker code while running. The CodeSource carries either inline bytes or a URL.

// Inline content
await workManager.LoadCode(workerId, new CodeSourceContent(Encoding.UTF8.GetBytes(newCode)));

// From URL
await workManager.LoadCode(workerId, new CodeSourceUrl(new Uri("https://example.com/new-worker.py")));

What happens: 1. Fetches code from URL (if applicable) or uses inline content directly 2. Calls worker.engine.LoadCodeAsync(blob) 3. Adds history entry 4. Updates persisted configuration

Deletion

DeleteWorkerAsync

Permanently removes a worker:

await workManager.DeleteWorkerAsync(workerId);

What happens: 1. Removes worker from in-memory registry 2. Deletes worker config from state store 3. Removes Dapr topic subscription 4. Does not stop the worker first (subscription is removed regardless)

Recovery

Worker recovery is automatic on service startup.

The WorkManagerRecoveryHostedService calls RecoverWorkersAsync() inside a BackgroundService before Kestrel begins serving traffic. The health check reports Degraded until recovery completes, and Unhealthy if recovery fails.

How recovery works

  1. On startup, the hosted service automatically restores all workers from the Dapr state store
  2. For each worker:
  3. Creates a new engine instance via the engine factory
  4. Loads the original code into the engine
  5. Re-establishes the Dapr pub/sub subscription
  6. Sets status to Running
  7. After recovery, the health check reports Healthy and Kestrel starts serving traffic

Multi-instance safety (leader election)

When multiple WorkManager instances start concurrently (rolling updates, HPA scaling, pod rescheduling), only one instance performs recovery — the others would otherwise load every worker's engine into RAM, wasting memory and CPU (see WM #4 — the N×W engine-load problem).

The RecoveryLeaderElector acquires a CAS lease on the Dapr state store key workmanager/recovery/leader (TTL WORKER_RECOVERY_LEASE_TTL_SECONDS, default 5 minutes). The instance that wins the lease performs recovery; the others mark recovery complete locally and rely on the normal message-flow path. If the leader crashes mid-recovery, the lease auto-expires and another instance can take over after WORKER_RECOVERY_ACQUIRE_TIMEOUT_SECONDS.

The lease is released in a finally block when the leader finishes (success or failure), so the next scale-up cycle acquires it promptly rather than waiting for the TTL.

Horizontal scalability (competing consumers)

Multiple WorkManager replicas that share the same Dapr app-id act as competing consumers — Dapr "s consumer group semantics deliver each message on a shared topic to exactly one instance. Instances do NOT load-share: messages delivered to an instance without a matching worker are dropped (TopicResponseAction.Drop).

Operators MUST NOT deploy multiple WM replicas expecting throughput scaling on a single topic. The competing-consumer model ensures exactly-once delivery at the process level, not thread-pool parallelism. Topic-level throughput scaling requires partitioning messages across distinct topics.

Disabling automatic recovery

Set WORKER_RECOVERY_ON_START=false to opt out. The health check still reports Healthy (operators have explicitly chosen this mode), and workers must be restored manually via the RecoverWorkers gRPC endpoint.

Error handling

  • Recovery failures are surfaced as structured Error-level log events with per-worker detail
  • If recovery fails entirely, the health check reports Unhealthy and the pod will not become ready
  • Individual worker load failures are logged but don't stop recovery of other workers
  • Workers can still be recovered manually via the RecoverWorkers gRPC endpoint if needed

Known limitation: if the state store contains a corrupted worker entry that always fails to load (e.g. a stale nupkg URL that 404s, or a code blob whose SHA-256 no longer matches), the pod is permanently unready — there is no admin endpoint to delete a single bad state-store entry without clearing the entire store. A WORKER_SKIP_FAILED_WORKERS=true flag (skip unrecoverable workers, log + continue) is planned as a remediation.

Monitoring

The /health endpoint includes a recovery health check that reports: - Healthy — all workers recovered successfully (or recovery disabled by config, or another instance is the recovery leader) - Degraded — recovery still in progress (during startup) - Unhealthy — recovery failed (pod will not become ready)

History Tracking

Each worker maintains a history of code changes:

var history = workManager.GetWorkerHistory(workerId);
foreach (var entry in history)
{
    Console.WriteLine($"Code loaded at {entry.CreatedAt}");
    Console.WriteLine($"  Type: {entry.CodeSource.GetType().Name}");
}

What's tracked: - Initial creation with original code source - Each LoadCode call

What's not tracked: - Runtime execution errors (these go to logs) - Message processing counts

Cleanup on Disposal

When WorkManager.DisposeAsync() is called:

  1. All Dapr subscriptions are disposed
  2. All topic-specific subscriptions are disposed
  3. In-memory registry is cleared

Note: State store data is NOT deleted. Workers can be recovered on restart.

Best Practices

  1. Recovery is automatic: Worker recovery runs automatically on startup via the WorkManagerRecoveryHostedService. No manual gRPC call is required. The service readiness probe ensures traffic is not routed until recovery completes.

  2. Track history for debugging: History entries help audit code changes: 

    var history = workManager.GetWorkerHistory(id);

  3. Stop before delete is optional: DeleteWorkerAsync cleans up subscriptions regardless of state, so explicit StopWorkerAsync is not required before deletion.

  4. Code updates don't restart subscriptions: LoadCode updates code in-place without disrupting processing.