Asynchronous operations and polling

Moltaro has both request-blocking operations and durable background work. A human developer or coding agent must identify the execution model before invoking an operation: a successful enqueue response means that work was accepted, not that its business effect has completed.

Do not infer execution from C# or HTTP syntax

An async C# method or a Task<T> return type can still execute while the HTTP caller waits. Conversely, Moltaro enqueue endpoints currently return an HTTP 200 envelope containing a queued resource. Neither the word async nor the HTTP status alone determines the lifecycle. Use the installed OpenAPI schema, Function Catalog metadata, binding or publication kind, and the returned Status plus polling URL.

Every route below resolves against the configured WORKSPACE_API_BASE_URL, not moltaro.com. The function catalog, schedules, and operations routes are supported administrative APIs of the configured Moltaro Application. They are not Portal administration APIs.

Execution-model decision table

Surface	Machine-readable signal	Execution model	Completion evidence
Entity validation or before-save mutation	Function Catalog CurrentPublishedContract is Validation or BeforeSaveMutation; binding Source is EntitySave	Request-blocking inside the record save	The original record API response
Command function	API publication Kind is Command; catalog contract is Command	Request-blocking	Command response contains the terminal run result
HTTP endpoint function	Catalog contract is HttpEndpoint	Request-blocking for that inbound HTTP request	The endpoint’s HTTP response; any job it explicitly enqueues is separate work
Global job through API	API publication Kind is Enqueue; catalog contract is Job	Queued worker execution	Poll the returned JobId
Function schedule	Schedule targets a published global Job function	Each due CRON occurrence is enqueued for a worker unless the target function already has queued or running work; an overlap is recorded as Skipped	Query jobs and runs by ScheduleId
Entity trigger	Binding Source is EntityTrigger; catalog contract is TriggerHandler when the function is trigger-only	The record commits first, then a worker handles the trigger	Query jobs and runs by TriggerBindingId or correlation id
Entity or global UI action	Catalog contract is Action; runtime route ends in /enqueue	Validation is request-blocking; execution is queued	Observe the active action job and Function operations
Boards or Entitlement C# application automation call	developer-surface entry has ServiceKind = ApplicationAutomation; enclosing function contract must be Action, TriggerHandler, Command, Job, or HttpEndpoint	The facade call waits for its own separate main application transaction	Successful MoltaroRuntimeResult; poll only ids actually returned in FollowUpOperationIds
Net Operation Project Check or Build	POST .../net-operation-project/builds returns a build record with Id and Status	Queued build worker	Poll GET .../builds/{buildId}
Workspace UI Check or Build	POST .../ui-project/builds returns a build record with Id and Status	Queued build worker	Poll GET .../builds/{buildId}

Read enum names from x-enum-varnames in the installation-local OpenAPI document instead of guessing numeric values. For the current contract, BusinessFunctionContract includes Validation = 0, BeforeSaveMutation = 1, Action = 2, Job = 3, TriggerHandler = 5, HttpEndpoint = 6, and Command = 7. BusinessFunctionApiPublicationKind is Command = 0, Enqueue = 1; BusinessFunctionBindingSource is EntitySave = 0, EntityTrigger = 1.

Discover the function before invoking it

For an authoring or administrative client, use the Configuration API rather than class names or UI labels as the source of truth:

GET /api/workspace/admin/function-catalog
GET /api/workspace/admin/function-catalog/{functionId}
GET /api/workspace/admin/function-catalog/binding-summaries
GET /api/workspace/admin/api-functions
GET /api/workspace/admin/function-schedules

Join these responses by BusinessFunctionId:

1. CurrentPublishedContract says which runtime contract is active now.
2. A binding Source distinguishes an in-save binding from an after-commit trigger.
3. An API publication Kind distinguishes synchronous command invocation from asynchronous enqueue publication. IsEnabled and IsRetired say whether the published key is callable.
4. A schedule identifies its target BusinessFunctionId, Status, NextRunAt, LastStatus, and target health. An enabled schedule enqueues only future due occurrences; creating or updating it does not run the function immediately.

Source-language inspection helps while authoring, but the catalog after a successful Build is the runtime truth. Preserve the stable function key across revisions.

CRON schedule: exact asynchronous flow

A Function Schedule contains configuration, not executable code. Its target must be a published global function with the Job contract. At a due time:

enabled schedule becomes due
  -> scheduler enqueues one job with Source = Schedule and ScheduleId
  -> worker leases that job
  -> worker creates and durably completes a function run
  -> job reaches Completed only after terminal run success is confirmed
  -> scheduler keeps the next planned occurrence

The record or API call that originally caused the schedule to be created is not held open. The scheduled run uses the system automation actor, and there may be no original user. Active-work deduplication skips a new occurrence while the previous job for the same schedule is still Queued or Leased.

To verify one schedule, read it and then filter the operations APIs by its id:

GET /api/workspace/admin/function-schedules/{scheduleId}
GET /api/workspace/admin/function-operations/jobs?Source=Schedule&ScheduleId={scheduleId}&Limit=50
GET /api/workspace/admin/function-operations/runs?Source=Schedule&ScheduleId={scheduleId}&Limit=50

NextRunAt shows planning, a job row proves enqueue, and a terminal run proves execution outcome. LastStatus on the schedule is a summary, not a substitute for the job or run detail when diagnosing failure. Moltaro does not publish Completed for an occurrence while its linked run is Running or otherwise unconfirmed. After a restart between the two metadata writes, recovery uses the persisted run outcome without re-executing the business function: it completes an already successful run, preserves an accepted cancellation as Cancelled, or restores an ordinary terminal failure to Queued with its configured retry delay until the attempt limit is reached. Incomplete, ambiguous, protected, and exhausted terminal outcomes fail closed instead of being redelivered.

Function job lifecycle

An API-enqueued function returns a status model containing JobId. Retain that id and poll:

POST /api/workspace/functions/{functionKey}/enqueue
GET  /api/workspace/functions/jobs/{jobId}
POST /api/workspace/functions/jobs/{jobId}/cancel

When the publication declares MoltaroApiEnqueue.TimeoutSeconds, the enqueue response and later status reads expose that captured value. It belongs to the job, not to one attempt: retries keep it even if a later Build changes the publication or an administrator changes the runtime Job default. A null value means the runtime default remains in effect.

Job status values are:

Status	Value	Meaning	Terminal
Queued	0	Waiting for a worker or for NextRunAt	No
Leased	1	Claimed by a worker	No
Completed	2	Worker execution completed successfully	Yes
Failed	3	Execution ended in error	Yes
Cancelled	4	Work was cancelled	Yes

Completed is published only after the linked run is durably Success or Skipped with CompletedAt, DurationMs, and a non-empty ResultSummary. On Completed, inspect Run.Status, Run.ResultData, and Run.ResultSummary; on Failed, inspect LastError and Run.ErrorSummary. A successful status read containing Failed is not a successful business operation. Call the public cancel endpoint only while its response says CanCancel: true; for API-enqueued jobs this is the queued state.

Administrators use Function operations to stop an entire retry chain:

POST /api/workspace/admin/function-operations/jobs/{jobId}/stop-retries
{ "ExpectedModifiedAt": "<the ModifiedAt returned by Function operations>" }

The job projection exposes CurrentAttempt, EffectiveMaxAttempts, RemainingAttempts, RetryDelaySeconds, NextRunAt, WillRetry, CanStopRetries, RetryStopState, and stable reason codes. A queued or expired leased job is finalized immediately. A live leased attempt is cancelled cooperatively and cannot create a later attempt; code that ignores cancellation may still finish or time out. A fully durable success for that exact current attempt wins and produces Completed. Repeating an already accepted command is idempotent; a stale version for a different active state returns a conflict. CancellationReasonCode: "job.retry.stopRequested" is the stable canonical reason for an accepted administrative retry-chain stop. It is persisted by the stop command; projections also normalize upgraded active jobs whose legacy cancellation intent did not originally store a code. Disabling or deleting a schedule prevents new occurrences but does not silently cancel an already-created job. Schedule responses keep its active job and retry-control state visible. Use the explicit Stop retries action when that job must receive cooperative cancellation and be prevented from starting a later attempt; it remains visible in Function operations afterward.

Use DeduplicationKey for repeat submissions of the same active operation. It suppresses duplicates only while an existing job with that key is Queued or Leased; it is not permanent business idempotency. Keep the returned job id across transport failures and client restarts instead of enqueueing a replacement because polling is slow.

Polling algorithm

Use the same bounded pattern for function jobs and project builds:

1. Submit once and persist the returned job or build id.
2. Poll only that resource, using a modest delay and an overall client deadline. Honor server retry guidance if a response provides it.
3. Continue while the status is non-terminal. Function jobs use Queued and Leased; project builds use Queued and Running.
4. Treat Completed or Succeeded as terminal success and inspect the result or artifact. Treat Failed and Cancelled as terminal non-success and read stored diagnostics.
5. If the client deadline expires, report the id and latest server state. Do not submit a duplicate automatically.

Both project build status enums currently use Queued = 0, Running = 1, Succeeded = 2, Failed = 3, and Cancelled = 4. Build-kind values are different: Net Operation Project uses Build = 0, Check = 1, while Workspace UI uses Check = 0, Build = 1.

Net Operation Project Build responses have an additive nullable Failure object. When present, branch on its stable ReasonCode, not on the localized summary. It also carries category, retryability, a validated PostgreSQL SqlState when available, a safe logical target, and a stable remediation code. The Build id is the correlation id used by Studio and Function Operations. storage_capacity_exhausted (SQLSTATE 53100 when raised by PostgreSQL) means an operator must free or expand storage before submitting a new Build. publication_commit_unconfirmed means the server could not prove the exact Build/artifact/project commit and did not replace the previous active artifact. An exact replay of a confirmed published Build is write-free.

For a module automation facade, the returned Task covers the facade’s main transaction; it does not imply that durable outbox delivery has finished. FollowUpOperationIds contains only follow-up work that was actually queued. Poll a returned function job id at GET /api/workspace/functions/jobs/{jobId}. If the list is empty, do not invent a job or enqueue the operation again; verify the synchronous result and the related audit/resource activity instead.

Run history is different from queue state

A job describes delivery and worker state. A run describes one actual function execution and its business outcome. A retry can update the job’s attempt count and create another execution record, so agents must not collapse the two concepts.

Use the runtime job endpoint for a job created through public API enqueue. Use the Configuration API’s Function operations routes for administrative diagnostics across schedules, triggers, actions, test invocations, commands, and endpoint runs. Filter by FunctionId, Source, ScheduleId, TriggerBindingId, ActionId, entity ids, user ids, correlation id through search, time range, and status as declared by the installed OpenAPI document.

Authoring checklist

Before handing off asynchronous logic, verify all of the following:

• the active Function Catalog contract and binding/publication kind match the intended execution model;
• a scheduled target is a global Job function and the schedule is enabled, healthy, and has the expected NextRunAt in the effective time zone;
• the caller stores job/build ids, uses bounded polling, and distinguishes terminal failure from transport success;
• retries reuse the supported deduplication or business idempotency key;
• cancellation follows server-provided CanCancel and does not assume that stopping polling stops server work;
• operations and business audit/ledger history are checked after execution.

See Function schedules, Commands and API functions, Operations and diagnostics, and Reliable API automation for the surface-specific contracts.