Skip to content
MoltaroMoltaroMoltaro Docs

Operations and diagnostics

Every business-function execution in Moltaro leaves an operational record. Two concepts carry that record: the run history (a persisted row for every execution, whatever its source) and the job queue (queued work items for executions that run in the background). This page explains both, how to cancel work, and where the diagnostics live in the product and in the admin API.

For the machine-readable rules that tell a client whether an operation is request-blocking or queued before it is invoked, including the exact CRON schedule flow, see Asynchronous operations and polling.

Run history

Section titled “Run history”

Every execution writes a run row when it starts and finalizes it with the outcome. A run stores the function id and version, the execution source (command, action, trigger, schedule, HTTP endpoint), the actor and original user, a correlation id, the duration in milliseconds, snapshots of the invocation arguments and the result, optional trigger/schedule and entity references, error type and message, a log summary, and counters for changed records and validation errors.

A run always ends in one of nine statuses:

  • Success — the function run finished successfully.
  • ValidationFailure — the function returned validation issues that rejected the operation.
  • PermissionFailure — the run was refused because the actor or function lacked a required permission or capability.
  • Timeout — the run was aborted because it exceeded its allowed execution time.
  • RuntimeException — the run ended with an unhandled exception thrown by the function code.
  • Cancelled — the run was cancelled before it completed.
  • Running — the run is currently executing and has no final outcome yet.
  • Skipped — the function chose not to act for this invocation and made no changes.
  • StoppedByWorkerRestart — the run was interrupted because the worker host restarted while it was executing.

Runs triggered by HTTP endpoints carry an additional request/response detail: endpoint identity and route, request method and path, response HTTP status, and redacted header/query summaries. Raw request bodies, secrets, and authorization material are never persisted.

Job lifecycle

Section titled “Job lifecycle”

Background executions go through a queued work item — a job — before a worker picks them up. A job tracks its attempt count, next attempt time, lease expiry, and last error, and moves through five statuses:

  • Queued — waiting in the queue for a worker to pick it up.
  • Leased — claimed by a worker under an active lease.
  • Completed — finished executing successfully.
  • Failed — execution finished with an error.
  • Cancelled — cancelled before it could complete.

Jobs can carry a deduplication key. The key is an active-work guard, not a permanent business key: it suppresses a duplicate enqueue only while an existing job with the same key is Queued or Leased. Terminal jobs (Completed, Failed, Cancelled) keep their key for history and diagnostics but never block a future trigger or schedule occurrence from enqueueing new work with the same stable key.

A job can become Completed only after its latest linked run is durably terminal with Success or Skipped, CompletedAt, DurationMs, and a non-empty ResultSummary. Summaryless successful results receive the deterministic summary Completed. (Skipped. for skipped results). If any part of that confirmation is missing, Moltaro fails the job closed instead of exposing a false successful schedule occurrence.

Recovery reconciles both sides of an interrupted metadata handoff. A run that remains Running after its job is terminal is closed deterministically, and a false Completed job is corrected to Failed. Conversely, when the run is already durably successful but the job is still Queued or Leased, recovery completes only the job metadata. It never invokes the business function again. An active job on another worker, or a job still registered in the current worker process, continues to protect its run from duplicate execution.

Cancelling work

Section titled “Cancelling work”
  • A queued job is cancelled directly; if the function is already running, the same call requests cancellation of the active execution.
  • A running run can be cancelled through its run id when the function supports manual cancellation; a non-cancellable run is rejected with a conflict.

Both operations are written to the workspace audit log.

Runtime settings

Section titled “Runtime settings”

Workspace-level runtime settings control run-history retention. The default retention mode is Forever: the runtime never deletes run history unless an administrator explicitly switches the mode to Days and supplies a retention day count. The purge then deletes only terminal run rows older than the calculated cutoff, in bounded batches. Non-terminal runs and all runs linked to active queued or leased jobs are intentionally retained so cleanup cannot erase execution-fence or unconfirmed-success evidence. Queue records, function definitions, versions, bindings, and schedules are never touched. Settings updates use optimistic concurrency through RowVersion and require the Admin role.

Where to look in the product

Section titled “Where to look in the product”
  • Administration, group Monitoring: the Function operations page shows the queue overview, run history, job list, and cancellation actions; the API Functions page monitors functions exposed for direct API invocation; the HTTP Endpoints page monitors inbound webhook endpoints.
  • Constructor, group Automation & logic: the Function Catalog page links from each function to its run history.

Function operations API

Section titled “Function operations API”

Diagnostics are also available through the Configuration API under /api/workspace/admin/function-operations. The published read and cancel endpoints require configuration-management access. Runtime settings are managed from the product UI and are not part of the curated API reference.

GET  /api/workspace/admin/function-operations/overview
GET  /api/workspace/admin/function-operations/runs
GET  /api/workspace/admin/function-operations/runs/page
GET  /api/workspace/admin/function-operations/runs/{runId}
POST /api/workspace/admin/function-operations/runs/{runId}/cancel
GET  /api/workspace/admin/function-operations/jobs
GET  /api/workspace/admin/function-operations/jobs/page
GET  /api/workspace/admin/function-operations/jobs/{jobId}
POST /api/workspace/admin/function-operations/jobs/{jobId}/cancel

The overview endpoint returns queue counts, recent runs, and failed or retrying queued runs. Run and job lists filter by function, status, source, trigger binding, schedule, date/time range, action, entity definition and instance, function version, actor and original user, and free-text search; the /page variants add paging and sorting, and the job lists can be narrowed to retrying jobs. For how executions reach the queue in the first place, start from the business logic overview and schedules.