Namespace Virtufin.WorkManager.Services

Classes

ApiLifecyclePublisher: Publishes worker lifecycle events to sys.workmanager.worker.lifecycle (ce-type com.virtufin.workmanager.worker.lifecycle, subject worker/<id>, the state in the payload's state field; names from Virtufin.WorkManager.Engines.WorkManagerNames). Events are CloudEvents v1.0 envelopes with the WorkManager's URN as the source. The publisher is best-effort: failures are logged but never propagated, so an unreachable API does not break the WorkManager's worker state machine.

AppMetrics: Application-level metrics for the WorkManager service.

DaprCircuitBreakerHealthCheck: Health check that reports Dapr circuit breaker state.

DaprResiliencePipeline: Provides resilience policies for Dapr operations with retry and circuit breaker. Constants are configurable via DaprResilienceOptions (env-var-driven).

DotNetDllBridgeWarmupHealthCheck: Health check that verifies the DotNetDll engine's CoreCLR bridge warmup has completed, so a pod is not reported Ready until the one-time, potentially tens-of-seconds-long bridge bootstrap is off the request path (see HostFxrBootstrap).

Hosted service that eagerly warms up the DotNetDll engine's CoreCLR bridge on startup (Virtufin.WorkManager.Engine.DotNetDll.HostFxrBootstrap.EnsureInitialized()), so the one-time bootstrap cost is paid before this pod is reported Ready rather than on a live worker's CreateWorker call.

Why this matters: under NativeAOT, loading a JIT-compiled worker DLL requires bootstrapping a secondary CoreCLR runtime via libhostfxr the first time any pod does this in its process lifetime. That cost has been observed live to take tens of seconds -- far longer than Virtufin.WorkManager.Engine.DotNetDll.HostFxrBootstrap's own documented ~200-500ms expectation -- comfortably busting the API Gateway's flat, 30-second deadline on forwarded dynamic-dispatch calls. Paying this cost during startup instead removes it from the request path entirely.

Kestrel readiness is gated via DotNetDllBridgeWarmupHealthCheck: Unhealthy while warmup is in progress, Healthy once it completes (whether it succeeded or failed -- see FailWarmup(Exception)).

DotNetDllBridgeWarmupState: Tracks whether the DotNetDll engine's CoreCLR bridge has been warmed up (or an eager warm-up was attempted and failed) on startup. Used by the bridge warmup health check to gate Kestrel readiness.

HostFxrDotNetDllBridgeWarmup: Thin seam around Virtufin.WorkManager.Engine.DotNetDll.HostFxrBootstrap.EnsureInitialized() so DotNetDllBridgeWarmupHostedService's success/failure handling is unit-testable -- the real bootstrap always succeeds as a fast no-op in a JIT test host (Virtufin.WorkManager.Engine.DotNetDll.HostFxrBootstrap.IsCoreClrLoaded() is true there), so a test double is needed to exercise the failure path.

LeaderLease

OwnedTriggerCleaner

RecoveryHealthCheck: Health check that verifies worker recovery has completed successfully.

RecoveryLeaderElector

RecoveryState: Tracks the state of worker recovery on startup. Used by the recovery health check to gate Kestrel readiness.

ResilientDaprPublisher: Buffers outbound pub/sub messages during Dapr outages and flushes on reconnect.

RuntimeWorkerLoadCoordinator

Asynchronous worker creation (worker-management spec, "Create returns before the code is loaded"). CreateWorker registers a Runtime-backed worker as Loading and returns; this coordinator then drives the load in the background (LoadAsync(CancellationToken)), commits the outcome to the worker's catalog record, starts it if a StartWorker arrived meanwhile, and publishes the loaded / load_failed lifecycle event.

Deferred start without a lost update. The outcome commit and TryRequestStartAsync(Guid, bool, CancellationToken) both go through etag compare-and-swap on the same catalog record, from any gateway replica. The commit reads StartRequested under the etag it writes with, and a start request only lands on a record still Loading. So whichever writes second sees the other's write, and a start is never dropped. Starting is idempotent, so a retried commit that starts twice is harmless.

At startup it also re-drives any Runtime-backed worker left Loading (a gateway replica that died mid-load). Every replica does this; LoadAsync(CancellationToken) is idempotent and only one commit can win the compare-and-swap, so the others just observe the outcome.

RuntimeWorkerPokeService

Periodically pokes the Runtime actor of every Runtime-routed worker whose persisted status is running, so a worker that has gone quiet never ends up permanently deaf (issue #47).

Runtime topic subscriptions live in the Runtime service's in-process RuntimeTopicSubscriptionPool and are (re)created only by StartAsync or the actor's OnActivateAsync self-heal -- and a Dapr actor activates only when something invokes it. For a Running worker with no inbound traffic, nothing ever does: no subscription -> no messages -> no invocations -> no reactivation. Two failure modes fall out of that: a Runtime pod restart (every subscription it hosted is gone; the gateway's own recovery deliberately doesn't touch Runtime workers since 0.16.1, and WorkerHeartbeatService only touches locally-registered workers, which Runtime-routed ones never are), and a quiet worker exceeding the Runtime's 24h ActorIdleTimeout.

A live actor gets a cheap no-op Touch (and its idle timer resets); a deactivated one reactivates -> the self-heal reloads code and resubscribes its topics if the Runtime record says Running -- the exact tested path that already exists. The gap between "worker stops receiving" and "worker receives again" is therefore bounded by this interval instead of being unbounded. Interval comes from IntervalSecondsEnvVar (default 10 min, floored at 30s -- comfortably under the 24h idle timeout even at the floor).

WorkManagerGrpcService

WorkManagerRecoveryHostedService

Hosted service that automatically recovers workers on application startup, gated by Enabled and coordinated across multiple WorkManager instances via RecoveryLeaderElector.

When multiple instances start concurrently, only the leader performs recovery — the others wait for the leader lease to expire (or be released) before they would attempt to take over. This avoids the N×W engine-load waste described in WM #4.

Kestrel readiness is gated via RecoveryHealthCheck (mapped on the /ready endpoint, not /health -- see Program.cs):

Unhealthy while recovery is in progress, or if it failed.
Healthy once recovery completes (whether this instance performed it or another leader did).

WorkerHeartbeatService: Periodically touches the actor for every worker registered locally on this instance, so a running worker's actor never gets idle-deactivated out from under it. See TouchAsync(CancellationToken)'s doc comment for why this is needed: message dispatch for a running worker happens entirely through WorkManager's topic-subscription pool, never through an actor method call, so a worker actively processing messages can still look idle to Dapr's actor runtime.

WorkerLoadFailedException: A worker whose background load failed (async CreateWorker).

WorkerProcessingRetryPipeline: Polly-based retry pipeline for worker business-logic processing failures, as distinct from DaprResiliencePipeline (which wraps Dapr/Gateway infra calls). Exhausting all attempts lets the caller's exception propagate so it can dead-letter the message -- see WorkerProcessingRetryOptions for why this exists as its own pipeline.

Interfaces

IDotNetDllBridgeWarmup: Thin seam around Virtufin.WorkManager.Engine.DotNetDll.HostFxrBootstrap.EnsureInitialized() so DotNetDllBridgeWarmupHostedService's success/failure handling is unit-testable -- the real bootstrap always succeeds as a fast no-op in a JIT test host (Virtufin.WorkManager.Engine.DotNetDll.HostFxrBootstrap.IsCoreClrLoaded() is true there), so a test double is needed to exercise the failure path.

IOwnedTriggerCleaner: Deletes the triggers a worker owns (owner_worker_id, api 0.8.6+) when the worker is deleted, so a trigger never outlives the worker it drives.

IRecoveryLeaderElector

IWorkerRecoveryExecutor: Abstraction over the WorkManager's recovery entry point. Allows the hosted service to be unit-tested without mocking the sealed WorkManager class.

Table of Contents

Namespace Virtufin.WorkManager.Services

Classes

Interfaces