On-premise operations

Use on-premise operations when you are responsible for running a Moltaro installation in a controlled environment.

Operations work covers deployment, configuration, updates, backups, restore procedures, monitoring, maintenance, and troubleshooting. The goal is simple: keep the governed workspace available, recoverable, and understandable to the team that owns it.

Breaking Entity Security upgrade

The Security Statements cutover does not translate old Entity grants, field rules, record assignments, or inherited-access state. It preserves Entity Definitions, fields, business records, non-security configuration, Responsibility Groups, and memberships. Owner and Admin no longer receive implicit Entity record access.

Before upgrading, stop writes and take a verified database and file-storage backup. After migration, open every used Entity Definition and configure its three Security sections: Responsibilities, Access Policies, and Permissions. Create the required policies and review each server Plan before applying it atomically. Ordinary fields inherit their root permissions; use Permissions only for deliberate field refinements. At minimum, every actor who must read records needs a working root View binding. Definitions without one fail closed.

Before reopening production, verify representative actor access, record lifecycle, import/export, comments, attachments, tags, activity/history, responsibilities, and reference previews. If verification fails, restore the complete pre-upgrade backup. Rolling application binaries back without restoring the database is not sufficient because the old security schema and data were removed.

Runtime VM capacity and diagnostics

Shared and Dedicated Runtime VMs use the same placement and diagnostic model. Every VM has an explicit maximum application count and one stable diagnostic profile for every managed application. The profile does not divide the host into preallocated pieces and does not impose CPU, memory, swap, or PID limits. Its memory warning and critical thresholds are monitoring values only.

The maximum application count is an operator-owned placement value, not a limit calculated by Portal. Portal projects that count times the application memory warning threshold. If the projection exceeds physical RAM, Portal shows an advisory capacity warning but still allows VM provisioning. There is no synthetic host reserve or per-application resource cap. Use Runtime Health to observe real pressure and reduce density or choose a larger VM when workloads compete.

The maximum application count and diagnostic thresholds remain editable after applications are assigned. Runtime health, diagnostic capability, and profile hash synchronization never block placement. If a VM has an available slot and is otherwise ready, Portal accepts the application and queues its deployment.

Runtime agent 0.1.29 and later place each application in a dedicated unbounded systemd slice and use that slice as Compose cgroup_parent for attribution. They do not write resource-control directives to the slice or Compose override. The override is preserved by moltaroctl upgrade.

Managed application runtime health

Application lifecycle and runtime health are separate signals. A running application can be Healthy, Warning, Degraded, or Unknown; a stopped application, maintenance window, or active lifecycle operation is Paused. Portal refreshes the Web, API, Worker, and RabbitMQ component measurements from the Runtime VM agent every 30 seconds. It shows aggregate app CPU and memory, the diagnostic memory warning/critical thresholds, and component process counts.

Sustained host CPU contention means applications are competing for execution time and users may notice slower requests or timeouts. Crossing a configured memory threshold or observing an unexpected component restart produces a diagnostic incident; it does not throttle, terminate, or block the application. Portal keeps recently resolved incidents so an operator can correlate a transient symptom with its cause.

Application owners see safe measurements, restart/OOM facts, effects, and a recommendation to contact their portal administrator or move the application to a Runtime VM with more capacity. Platform administrators additionally see the Runtime VM relationship, Docker container diagnostics, exit details, and the technical event timeline. Technical container identifiers and host diagnostics are never included in the customer response.

Entity search operations

Entity search runs in ordinary PostgreSQL record queries. Operators do not provision an index service, search-data volume, rebuild job, queue, or search-specific backup. The normal maximum configured path depth is controlled by Moltaro:EntitySearch:MaxPathDepth and defaults to 3; API and worker configuration should use the same value.

Managed Portal backups use the ordinary Moltaro artifact: PostgreSQL dump, uploaded files, and Data Protection keys. Green restore restores that artifact and migrates the database forward before starting the application. It does not restore a search directory or schedule a post-restore search rebuild.

Old search-index directories left by an earlier release are not read and are not removed automatically during upgrade.

File-body storage

A standalone or self-managed installation stores new file bodies in PostgreSQL by default. The operator can select Amazon S3 during a fresh database bootstrap with moltaroctl prepare-database --file-body-storage amazon-s3; selecting S3 also requires a configured provider implementation.

Portal-managed applications use a different, explicit contract. Portal creates an application-scoped object-storage binding, enables new uploads, and delivers separate short-lived signer capabilities to API and Worker. Runtime VMs never receive provider account credentials. Before a managed application is declared ready, the Runtime VM agent verifies that the binding is ready, new uploads are enabled, the workspace database selects Amazon S3, both runtime hosts enable managed object storage, and their binding identifiers and versions agree. Restore preserves descriptors for legacy database-backed files and switches new uploads to the restored application’s ready S3 binding.

Legacy database file bodies can be migrated without changing file references or descriptor identifiers. Run the following sequence with API, Web, and Worker fully stopped after the read-only plan:

moltaroctl file-storage legacy-to-s3 plan --install-root <workspace-root>
moltaroctl file-storage legacy-to-s3 run --install-root <workspace-root>
moltaroctl file-storage legacy-to-s3 verify --install-root <workspace-root>
moltaroctl file-storage legacy-to-s3 finalize --install-root <workspace-root>
moltaroctl file-storage legacy-to-s3 compact --install-root <workspace-root>

run is resumable and retains the original PostgreSQL body. verify streams every object back and checks its size and SHA-256. rollback is available until finalize; finalization is rejected without a complete zero-error verify report. compact requires an empty fs.FileBody table and uses VACUUM FULL, so the application must remain stopped and the filesystem needs temporary headroom. Migration reports contain counts, byte totals, checksums, and failed descriptor IDs, but never capabilities, provider credentials, or file content.

Backups, runtime packages, and reboot recovery

A retained managed backup depends on the runtime package version recorded with it. Do not remove an old runtime artifact while any retained backup references that version. Backup deletion is an explicit operator decision; release tooling does not remove backups automatically.

Production service units, reverse-proxy configuration, and deployment contracts belong in source control. Required signer instances are installed with systemctl enable --now and deployment verifies both enabled and active before readiness. Operators should prove recovery with real sequential reboots: first the Portal VM (database, reverse proxy, PublicSite, documentation, and signers), then the Runtime VM (Docker, PostgreSQL, Runtime VM agent, application containers, APIs, Workers, legacy reads, and a new S3 upload).

Source control reconstructs software and topology, not production data or secrets. Keep database backups, object-storage data, Data Protection key rings, license material, provider credentials, SSH material, and signer master secrets in a separately encrypted recovery bundle with a tested restore procedure.

Releasing a managed environment

Portal is released as part of the complete managed environment, not through a standalone Portal command. From a clean frozen release source, run:

dotnet run --project src/OnPrem/Moltaro.Deployctl/Moltaro.Deployctl.csproj -c Release -- `
  release execute `
  --profile <environment> `
  --mode ReplaceRuntime `
  --version <version> `
  --source <full-40-character-release-commit> `
  --confirm "REPLACE <version>"

The named profile contains non-secret topology and environment-variable references. Required credentials must exist only in the command process environment; they are not command arguments or release evidence. The command installs locked frontend dependencies, builds and type-checks Portal, regenerates the complete public documentation and both API references, publishes the self-contained PublicSite service, updates Runtime VM agents, and replaces all discovered Managed App runtimes. Portal database migrations run inside the new release before the production symlink changes. If migration fails, the existing release remains current and production is not restarted.

After the switch, smoke checks cover the public site, Portal, documentation, both llms.txt entry points, and the Runtime and Configuration API references.