← All Nacreon documentation   |   Product website ↗
Spiryra by Nacreon Security Certificate Lifecycle · Installation and Feature Guide
2026.10 documentation · development preview

1.1About this documentation

This guide describes how to install Spiryra, connect the certificate authority you already operate, and use every feature in the console: certificate lifecycle, agents and clientless deployment, orchestration, identity, monitoring, service accounts and operations. Spiryra is published by Nacreon Security at nacreonsecurity.com.

Nacreon Security ships the Nacreon suite, of which Spiryra is the certificate lifecycle product. The suite, its documentation and the support policy are published at nacreonsecurity.com; this guide is the manual for one product in it.

Who this is for

Platform and security engineers who run the Docker host, PKI administrators who own the external CA, and operators who issue, deploy and renew certificates day to day. Installation chapters assume shell access to the Docker host; feature chapters assume a signed-in administrator.

How the chapters are arranged

ChaptersWhat you do there
1–3Understand the runtime, install a profile, sign in and complete the first-run checklist.
4–5Connect an external CA, then import, issue, renew and revoke certificates.
6–7Deliver certificates to Linux and Windows targets, then automate the work with workflows.
8–10Discover and monitor, control who can do what, and route notifications.
11–13Operate the deployment, review the security model and call the API.
14–17Read the verification status, troubleshoot, and look up terms and source documents.

Conventions

  • Monospaced text is a console label, file path, environment variable or literal value.
  • Code blocks run on the Docker host from the repository root unless the heading says otherwise.
  • /api/… paths are API routes; most have an equivalent screen in the console.
  • Commands that change data or delete volumes are marked destructive.
Note: This edition is a development preview. The production profile is a hardened runtime configuration, not a production certification. Read 14 Verification and readiness before connecting live CA credentials.

Callout styles

Note: Background or a constraint you must respect.
Warning: An action that can lose data, break certificate delivery or expose a credential.
Tip: A shorter path, a default worth keeping, or a check that saves time.
Verify: The evidence that proves the step worked.

1.2Edition comparison and release status

Both editions use the same images and application code. They differ only in runtime configuration, data volume, ports and safeguards.

 Test profileProduction profile
PurposeEvaluation, UI work, connector and agent experiments, orchestration and RBAC testing.A hardened runtime against the organization's existing CA.
NetworkLoopback HTTP, or one explicit LAN address on a trusted lab network.Loopback HTTP behind your HTTPS reverse proxy; exact public origin.
CookiesSPIRYRA_LAB_INSECURE_HTTP_COOKIE=trueSecure cookies; the lab switch is refused.
Certificate authorityOptional disposable EJBCA Community CA for evaluation.External CA only. No local CA, no demo seed.
Administratoradmin / Testing123, reapplied on every deployment.Generated bootstrap credential, printed once; change after sign-in.
Data volumespiryra-test_spiryra-dataspiryra-production_spiryra-data
Compose projectspiryra-testspiryra-production

What “development preview” means

The Rust API and console build, the test stack runs, and the verification described in 14.1 was executed. It does not mean the product is certified for production certificate management. The release gate deliberately blocks until live EST/SCEP and public ACME runs and an independent security review are complete; the WinRM orchestration steps have since run on a real Windows endpoint (WINDOWS_LIVE_WINRM_E2E.md).

Warning: Do not connect live issuing credentials, and do not manage the certificates your organization depends on, until you have accepted the limits in 14.2 for your environment.

Source-of-truth documents

DocumentContains
NATIVE_RELEASE_STATUS.mdVerification evidence, known gaps, release classification.
DEPLOY_FRESH.mdProfile commands, manual env workflow, backup, rotation.
RELEASE_GATE.mdSigned staging and production gate contract.
SERVICE_ACCOUNTS.mdMachine credentials, scopes and the service-account API.

1.3Deployment architecture

The console and API run in separate containers. The browser never reaches the API container directly: nginx serves the single-page console and proxies /api/ to the API over the internal network.

Browseroperator console
TLS reverse proxyproduction only
nginxstatic console + /api proxy
Rust APIauth · RBAC · lifecycle
SQLite volumesclm · fleet · issuance
External CAEJBCA · Vault · ACME · AD CS · EST · SCEP
Agent / SSH / WinRMcertificate delivery
SMTP / webhooksalerts and escalation

Runtime boundaries

  • Web container: nginx on port 8080, bound to 127.0.0.1 by default. In production your proxy publishes HTTPS.
  • API container: internal port 9080, health-checked, never published.
  • Data: one Compose volume per profile holding SQLite databases, the optional keyring, the ACME webroot and deployment evidence.
  • Secrets: deploy/<profile>.env, mode 0600, never committed.
  • Vault: AES-256-GCM wrapping for connector credentials, retained leaf keys, SSH keys, WinRM passwords and notification secrets.

What the API talks to

TargetDirectionRequirement
External CAOutbound onlyHTTPS with a trust bundle you configure, or WinRM/Web Enrollment for AD CS.
AgentsInbound pollingAgent polls /api/agent/jobs; no inbound port is opened on the target.
SSH / WinRM targetsOutboundTCP 22, or WinRM 5985 (lab) / 5986 (HTTPS).
SMTP / webhooksOutboundHTTPS webhooks only; private and loopback targets blocked unless explicitly enabled.
Public ACMEInbound port 80/.well-known/acme-challenge/ forwarded to the web container.
Tip: Spiryra never needs an inbound connection to a managed server. Agents poll, and SSH/WinRM are initiated by the API — which is why firewall rules stay simple.

1.4Supported platforms and prerequisites

Docker host

RequirementWhyCheck
Docker Engine with Compose v2Runs the API, web console and optional test CA.docker compose version
Python 3 with argon2-cffiGenerates the local administrator Argon2id hash.python3 -m pip install --user argon2-cffi
opensslRandom secrets and test CA material.openssl version
curlHealth checks used by the deploy and verify scripts.curl --version
Node.js 18+ and Playwright (optional)Automatic Settings-route browser check in the test profile.npm install, npx playwright install --with-deps chromium
TLS reverse proxy (production)HTTPS termination for the public origin.nginx, Caddy, HAProxy or your standard ingress.

Managed targets

PathSupported platformsRequirements on the target
Native agentLinux x86-64, Linux ARM64, Windows x86-64Outbound HTTPS to the console, a writable token path, and a service reload allowlist if you reload services.
Clientless SSHLinux and Unix-like hostsSSH user with write access to the certificate path and, optionally, a permitted reload command.
Clientless WinRMWindows ServerWinRM enabled with Auth\Basic on (Windows disables it by default); HTTPS (5986) with a trusted listener in production.

Certificate authorities

ConnectorMinimum on the CA side
EJBCA RESTEJBCA 7+ with the REST API enabled and an identity holding the CA, RA and end-entity-profile rules listed in 4.2.
HashiCorp Vault PKIVault with a PKI secrets engine and a signing role (lab-verified on 1.18.5).
ACME v2An RFC 8555 directory URL; EAB if your provider requires it; DNS API access for wildcards.
Microsoft AD CSA CA with a certificate template, reachable over WinRM or with the Web Enrollment role installed.
ESTAn RFC 7030 EST endpoint; the host must be listed in SPIRYRA_EST_ALLOWED_HOSTS.
SCEPAn RFC 8894 SCEP endpoint; the host must be listed in SPIRYRA_SCEP_ALLOWED_HOSTS.

Browsers

The console is a single-page application tested in current Chromium-based browsers and Firefox. It requires JavaScript and secure cookies in the production profile.

2.1Install the test profile

The test profile is the recommended first install. It deploys Spiryra with an optional disposable EJBCA Community CA, so you can exercise the whole lifecycle without touching a real authority.

Run the wizard

./scripts/deploy-test.sh

The wizard asks for the bind address, port and administrator username, then writes deploy/test.env with mode 0600, starts the stack, waits for health and prints the console URL and administrator credential.

What it creates

  • Spiryra on HTTP, loopback by default, or one explicit --bind address.
  • A disposable EJBCA Community CA from compose.test-ca.yaml, when enabled.
  • EJBCA client certificate, private key and ManagementCA bundle in deploy/test-ejbca/ (secrets mode 0600).
  • A paste-ready deploy/test-ejbca/connector-paste-values.txt for the Connectors screen.
  • A reusable deploy/test.env carrying SPIRYRA_TEST_CA=true, so later profile commands keep the CA container in the same lifecycle.

Flags

FlagUse it when
--checkValidate prerequisites and the existing env without touching containers.
--yesUnattended deployment; generated credentials are printed once.
--register-connectorCreate and test the EJBCA Test CA connector through the API instead of pasting values.
--no-ejbcaSpiryra only, no disposable CA.
--bind 192.168.40.204Lab access from another machine. HTTP only — never use this beyond a trusted network.
--skip-ui-checkThe host has no Node or Playwright.
--freshDestructive: replace test containers and the data volume.
--admin-password VALUEOverride the test administrator; also SPIRYRA_TEST_ADMIN_PASSWORD.

The test credential

Sign in as admin with Testing123 unless you overrode it. The password is reapplied after every test deployment, and deploy-production.sh refuses it. To reset a running stack without redeploying:

SPIRYRA_TEST_ADMIN_PASSWORD=Testing123 ./scripts/reset-test-admin-password.sh
Verify: curl -fsS http://127.0.0.1:8080/healthz returns ok, and curl -i http://127.0.0.1:8080/readyz returns 200. A fresh install may report degraded until a CA connector is configured — that is expected, not a failure.
Note: The test CA is disposable lab material. Never reuse its client key, PKCS#12 or issued certificates outside the lab, and never point a production deployment at it.

2.2Install the production profile

The production profile deploys Spiryra only. It never starts a CA and never seeds demo data; you connect the organization's existing CA after sign-in.

./scripts/deploy-production.sh \
  --origin https://spiryra.example.com

# preflight only, no containers touched
./scripts/deploy-production.sh --check

The origin must be an exact public HTTPS DNS origin: no path, query, IP address or localhost. The wizard generates a unique bootstrap administrator credential and prints it once.

What the preflight refuses

  • SPIRYRA_DEPLOYMENT_PROFILE not set to production.
  • A loopback, IP-address or non-HTTPS SPIRYRA_PUBLIC_ORIGIN.
  • SPIRYRA_LAB_INSECURE_HTTP_COOKIE=true, or any SPIRYRA_LAB_ALLOW_* switch.
  • The TLS-verification opt-outs SPIRYRA_ADCS_WEB_INSECURE=true or SPIRYRA_AGENTLESS_WINRM_INSECURE=true (also refused as 1).
  • SPIRYRA_METRICS_TOKEN missing, shorter than 32 characters, or still a template placeholder.
  • An env file readable or writable by group or world.
  • Migration tokens or a vault key shorter than 32 characters, or values that are not distinct.
  • The lab default password Testing123.
  • An API that advertises a local-ca kind or demoAvailable: true, or a Compose file such as compose.test-ca.yaml.

After the stack is healthy

  1. Put TLS in front of the console. Proxy your public origin to http://127.0.0.1:8080; see 2.4.
  2. Sign in with the bootstrap credential and change it immediately.
  3. Connect the existing CA under Connectors and test it before issuing anything.
  4. Review policy under Settings: expiry windows, default validity, key rules, SAN policy, session length.
  5. Configure notifications and run a backup; confirm /readyz is ready.
Verify: ./scripts/deploy-profile.sh production status reports healthy containers, and curl -i https://spiryra.example.com/readyz returns 200 with the CA connector check ok.
Warning: Do not publish port 8080. The console must only be reachable through the HTTPS proxy, otherwise session cookies and CSRF protection degrade to plain HTTP.

2.3Manual environment installation

Use this when you own the env file instead of letting the wizard generate it — for example when secrets come from your secret manager.

  1. Create the env file and lock it down.
    cp deploy/test.env.example deploy/test.env
    chmod 600 deploy/test.env
  2. Generate secrets. Three distinct migration tokens and one vault key:
    openssl rand -hex 32   # run three times for the migration tokens
    openssl rand -hex 32   # once for SPIRYRA_VAULT_KEY
  3. Generate the administrator hash.
    python3 scripts/create-lab-admin.py
    Paste the complete single-quoted SPIRYRA_LOCAL_ADMIN_ARGON2_HASH=… line it prints.
  4. Replace every remaining placeholder (REPLACE_WITH…) in the env file.
  5. Validate and start.
    ./scripts/deploy-profile.sh test check
    ./scripts/deploy-profile.sh test up
    ./scripts/deploy-profile.sh test status

Production follows the same pattern with deploy/production.env and the production profile. production check fails closed until the origin, cookies, tokens and vault key are valid.

Profile commands

./scripts/deploy-profile.sh <test|production> check
./scripts/deploy-profile.sh <test|production> up
./scripts/deploy-profile.sh <test|production> status
./scripts/deploy-profile.sh <test|production> digest
./scripts/deploy-profile.sh <test|production> down

start is accepted as an alias for up; both go through the same checks.

Stale images are refused, not served

A profile runs whatever images it was started from, and a container keeps its name and its Up (healthy) status while the sources move on. Each image is therefore stamped at build time with a SHA-256 over the files it is built from — the Dockerfile, the manifests, the sources, and for the API the agent binaries it serves — and the stamp is compared with the tree whenever someone asks.

status and doctor print the verdict after the container table and exit non-zero on drift; digest reports it on demand. up refuses to start images built from a different tree, because starting them is how the drift becomes serving, and up --allow-stale overrides that deliberately. An image built before the stamp existed cannot be compared at all, so it warns and starts instead of blocking an installation that predates the check.

./scripts/deploy-profile.sh production status   # non-zero when the images drifted
./scripts/deploy-profile.sh production build
./scripts/deploy-profile.sh production up      # refuses a drifted image
Verify: status ends with OK api and OK web lines and exits 0. After editing a source file it prints DRIFT with the digest the image was built from and the digest of the tree, and exits 1.

Generated material

PathContentsHandling
deploy/<profile>.envMigration tokens, vault key, bootstrap hash.0600. Back up with your secret manager, separately from data backups.
deploy/test-ejbca/Client certificate, key, ManagementCA bundle, PKCS#12.Disposable lab material only.
deploy/ca/Trust bundles mounted read-only into the API at /etc/spiryra/ca.PEM only; referenced by env, never uploaded through the UI.
backups/<profile>-<timestamp>/Verified SQLite snapshots plus a checksummed manifest.Contains sealed secrets but not the vault key.

2.4Reverse proxy and TLS

The web container serves HTTP on loopback. Your proxy terminates TLS for the public origin and forwards to it, preserving the browser's Host header.

Checklist

  • Terminate HTTPS for the exact SPIRYRA_PUBLIC_ORIGIN.
  • Proxy to http://127.0.0.1:8080; never publish that port.
  • Preserve Host (including a non-default port) and send X-Forwarded-Proto: https.
  • Forward /.well-known/acme-challenge/ on port 80 to the web container for public ACME HTTP-01.
  • Allow request bodies large enough for certificate imports and workflow payloads (a few MB is sufficient).

nginx example

server {
  listen 443 ssl;
  server_name spiryra.example.com;
  ssl_certificate     /etc/ssl/certs/spiryra.pem;
  ssl_certificate_key /etc/ssl/private/spiryra.key;

  location / {
    proxy_pass         http://127.0.0.1:8080;
    proxy_set_header   Host              $host;
    proxy_set_header   X-Forwarded-Proto https;
    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header   X-Real-IP         $remote_addr;
  }
}

# ACME HTTP-01 only
server {
  listen 80;
  server_name spiryra.example.com;
  location /.well-known/acme-challenge/ { proxy_pass http://127.0.0.1:8080; }
  location / { return 301 https://$host$request_uri; }
}
Note: The same-origin check compares the browser's Origin with the request Host. If the proxy rewrites Host to 127.0.0.1:8080, every write is refused with 403. Preserving Host is what keeps them in agreement.

TLS to external systems

Connector and webhook TLS is verified with the trust bundle you configure. Redirects are refused, embedded credentials in URLs are rejected, and no “skip verification” switch exists. Place private roots in deploy/ca/, mounted read-only at /etc/spiryra/ca, and reference the file by absolute path.

2.5Environment variable reference

All variables live in deploy/<profile>.env and are passed to the API container by Compose. Values marked secret are never printed by the console.

Core

VariablePurpose
SPIRYRA_DEPLOYMENT_PROFILEtest or production. Production refuses any other value.
SPIRYRA_PUBLIC_ORIGINExact browser-facing HTTPS origin. Used for links, cookies and same-origin checks.
SPIRYRA_DOCS_URLDocumentation link in the console sidebar footer. Defaults to /DEPLOYMENT_GUIDE.html, which the web image publishes beside the console. Set an https:// URL to point at your own copy.
SPIRYRA_METRICS_TOKENSecret. When set, /metrics requires Authorization: Bearer with this value. Required in production; with it empty the scrape endpoint answers unauthenticated to anything on the container network, so the preflight refuses a production env without one.
SPIRYRA_LOGIN_RATE_LIMIT / …_WINDOW_SECSSign-in attempts allowed per client address in one window. Default 10 per 300 seconds; 0 disables the bucket.
SPIRYRA_ENROLL_RATE_LIMIT / …_WINDOW_SECSSelf-enrollment attempts allowed per client address. Default 30 per 3600 seconds. Both windows are stored in the API database, so replicas that share that file spend one budget between them.
SPIRYRA_TRUSTED_PROXIESWhich peer may speak for the client. Unset or none uses the socket address and ignores X-Real-IP/X-Forwarded-For; * trusts the header from whatever connects; a comma-separated address or CIDR list (for example 10.0.0.0/8,192.168.40.24) trusts it only from those peers. The bundle defaults to the Docker bridge range, because the bundled web container is the only path to the API.
SPIRYRA_WINDOWS_PFX_KEEP_STAGED / SPIRYRA_WINRM_PFX_KEEP_STAGEDtrue keeps the staged PFX and password ciphertext after a verified import. Default: they are deleted, so private key material does not linger. A failed import always leaves them in place.
SPIRYRA_WEB_BINDOne explicit host address for the web container. Loopback by default; LAN binding is test-only.
SPIRYRA_LAB_INSECURE_HTTP_COOKIETest-only switch that allows cookies over HTTP. Refused in production.
SPIRYRA_VAULT_KEYSecret. AES-256-GCM wrapping key for every sealed value. Losing it makes sealed data unrecoverable.
SPIRYRA_VAULT_KEY_PREVIOUSSecret. Comma-separated previous keys accepted during rotation.
SPIRYRA_MIGRATION_ADMIN_TOKENSecret. Bootstrap administrator bearer used by scripts and recovery.
SPIRYRA_MIGRATION_READER_TOKENSecret. Read-only migration credential.
SPIRYRA_MIGRATION_REVIEWER_TOKENSecret. Review-only migration credential.
SPIRYRA_LOCAL_ADMIN_ARGON2_HASHSecret. Argon2id hash of the bootstrap administrator password.

Connectors

VariablePurpose
SPIRYRA_ACME_WEBROOTDirectory for HTTP-01 challenge files, shared with nginx.
SPIRYRA_EJBCA_ISSUED_CA_BUNDLEPEM bundle used to verify certificates returned by EJBCA. Empty disables issuance verification.
SPIRYRA_EJBCA_APPROVAL_POLL_SECSSeconds between polls of an EJBCA enrollment waiting for RA approval. Default 300, allowed 5–86400.
SPIRYRA_EJBCA_APPROVAL_MAX_POLLSFailed polls before that request is marked expired. Default 72, about six hours at the default interval.
SPIRYRA_EST_ALLOWED_HOSTSComma-separated EST hostnames. Empty keeps EST unavailable.
SPIRYRA_SCEP_ALLOWED_HOSTSComma-separated SCEP hostnames. Empty keeps SCEP unavailable.
SPIRYRA_AGENTLESS_WINRM_ALLOW_HTTPLab only. Permits WinRM over 5985. Production should use HTTPS 5986.

Agent plane

VariablePurpose
SPIRYRA_RUST_FLEET_SQLITEAbsolute path to the fleet database. Without it the agent plane answers 503 — there is no in-memory fallback.
SPIRYRA_AGENT_ENROLLMENT_TOKENSecret. ≥32-character campaign secret enabling self-enrollment.
SPIRYRA_AGENT_CAMPAIGN_TENANTTenant a campaign token enrolls into (lowercase identifier). Empty means the default tenant; an invalid value refuses enrollment rather than enrolling a host into the wrong inventory.
SPIRYRA_AGENT_CURRENT_VERSIONExpected agent build label, default go-preview.
SPIRYRA_AGENT_DOWNLOADS_DIRDirectory holding agent binaries served by the downloads route.
SPIRYRA_AGENT_RELEASE_CATALOGLocal JSON catalog of signed releases for staged upgrades.
SPIRYRA_APPROVED_SERVICESAgent-side allowlist of services a deployment may reload.
SPIRYRA_TLS_PROBE_ALLOWLISTAgent-side allowlist of TLS endpoints that may be probed. Empty denies all.
SPIRYRA_TLS_PROBE_MAX_TIMEOUT_SECSLocal cap on the requested probe timeout.
SPIRYRA_WINDOWS_PFX_SERVICE_ACCOUNTService identity granted read access to the imported PFX private key (SAM or UPN).
SPIRYRA_WINDOWS_PFX_REQUIRE_SERVICE_ACCOUNTWhen true, refuse a PFX import that has no configured identity.
SPIRYRA_UPDATE_SIGNING_PUBLIC_KEYPinned public key agents use to verify staged upgrade signatures.
Warning: Never change SPIRYRA_VAULT_KEY without the rotation procedure in 11.3. A new key alone makes every sealed credential and retained key unreadable.

2.6Upgrade an installation

  1. Read the change notes for the release, then back up the profile:
    ./scripts/backup-profile.sh production
  2. Check out the new source on the Docker host (same repository, same profile env file).
  3. Validate the configuration before replacing containers:
    ./scripts/deploy-profile.sh production check
  4. Deploy. The stack recreates the API and web containers; schema migrations run on startup.
    ./scripts/deploy-profile.sh production up
  5. Confirm readiness, then sign in and spot-check inventory, a connector test and one deployment target.
Verify: /healthz answers, /readyz returns 200 with the database and identity checks ok, and the console shows the expected certificate count.

Agent upgrades are separate

Agent binaries are upgraded through the signed release catalog, not by redeploying the control plane. See 6.4. An agent that is offline keeps its job queued; deployments do not replay automatically.

Rolling back an application upgrade

Redeploy the previous source revision. If a migration changed the schema, restore the pre-upgrade backup instead of downgrading binaries:

./scripts/restore-profile.sh production backups/production-20261031T020304Z --yes

2.7Remove a profile

./scripts/deploy-profile.sh test down
./scripts/deploy-profile.sh test down --volumes --yes   # destructive

down stops containers and keeps the data volume. Adding --volumes --yes removes the profile's containers and volume — all certificates, accounts, audit history and sealed credentials for that profile.

Warning: --volumes is irreversible. Take a backup first if the data has any value, and remember the backup is unreadable without the vault key.

For the test profile, ./scripts/deploy-test.sh --fresh also removes the disposable EJBCA containers and the ejbca-data volume. Production never has a test CA to remove, and profile commands never touch the other profile's data.

3.1Bootstrap credentials and sign-in

ProfileCredentialBehaviour
Testadmin / Testing123Prompted on every interactive deployment; reapplied after the stack is healthy. Lab only.
ProductionGenerated bootstrap passwordPrinted once at deployment. Change it after the first sign-in.

Sign-in flow

  • POST /api/auth/login returns a session cookie plus a CSRF token bound to that session.
  • Every browser-origin write must send the CSRF token; a missing or wrong value returns 403.
  • Bearer credentials — migration tokens, agents and API service accounts — are not subject to the browser CSRF rule.
  • Session lifetime follows the configured policy (sessionHours in Settings).
Note: If a single sign-on provider is enabled and marked default, local password sign-in may be hidden. Keep the bootstrap administrator reachable until SSO is proven — it is the recovery path.

First actions

  1. Change the bootstrap password under your account menu.
  2. Create one administrator account per person; stop sharing the bootstrap credential.
  3. Configure the CA connector, then policy, then notifications.
  4. Run a backup and store the vault key in your secret manager.

3.2First-run checklist

Work through this list in order. The first issued and deployed certificate is the fastest proof that the connector, policy, target and verification path all agree.

  1. Sign in and change the bootstrap credential (3.1).
  2. Connect the external CA under Connectors and press Test. Do not issue until the test passes (4.1).
  3. Set policy under Settings: expiry warning and critical windows, default validity, key rules, SAN policy, session length (5.7).
  4. Issue or import the first certificate and confirm it appears in inventory with the expected issuer and SANs (5.2).
  5. Register a deployment target — enrolled agent, SSH host or WinRM host — and confirm it is ready (6.1).
  6. Deploy the certificate to an absolute path with an optional approved reload and TLS endpoint (6.3).
  7. Verify presentation: the deployment only becomes verified when the fingerprint the socket serves matches the file (6.3).
  8. Close the loop: notifications, escalation, a profile backup and a green /readyz (10 Notifications, 11.2).
Tip: In the test profile, --register-connector creates and tests the lab EJBCA connector for you, so steps 2–4 can be proven before you touch a real CA.

3.3Password policy and account recovery

Policy

  • Local passwords are stored as Argon2id hashes; sessions store only a SHA-256 digest of the session token.
  • Policy covers minimum length and complexity, history, maximum age and lockout after repeated failures.
  • Policy is stored in the database and read by the sign-in path — a failed account-schema migration reports 503 instead of silently breaking sign-in.
  • Changing a password revokes existing sessions; an expired password is changed through the dedicated expired-password route.

Self-service recovery

Recovery requires an SMTP relay and a recovery email on the account. The administrator issues a one-time token, the user receives a link, and the token is single-use with a short expiry.

POST /api/auth/password/recover
POST /api/auth/password/change
POST /api/auth/password/change-expired
GET  /api/auth/password/policy

Administrators with the users.manage action can also reset a password directly, which is the fallback when mail is unavailable.

Verify: Configure SMTP first (10.1), then request a recovery for a test account and confirm delivery in the notifications history.
Note: Recovery never reveals whether an address exists: the response is the same for known and unknown accounts.

4.1Connector model and the credential vault

Spiryra is an interface to the certificate authorities you already operate. It never hosts an issuing CA in production. A connector is the configured, tested, audited path to one authority.

How a connector works

  • Non-secret settings (URLs, CA names, profiles, usernames) are stored as connector configuration and returned by the API.
  • Secrets — private keys, tokens, passwords, trust bundles — are sealed into one AES-256-GCM document bound to the connector id. The API returns only the names of stored secret fields, never the values.
  • Tests are explicit. A connector is not “working” until Test succeeds; the console shows the outcome and the audit log records it.
  • Issuance stays authoritative at the CA. Spiryra generates the key and CSR, then validates what comes back before anything enters inventory.

Add and test a connector

  1. Open Connectors → Add connector and choose the type. The form fields come from the connector specification (GET /api/connectors/specs).
  2. Fill the non-secret settings and paste or upload the secrets.
  3. Save, then press Test. The test authenticates, reads the CA, and reports what it found — for example the issuing CA and the available templates.
  4. Optionally run Pull profiles from EJBCA (or the equivalent discovery for that connector) to import the authority's template list for the issue form and CA policy.
Verify: The connector row shows a passing test with the issuing CA name and a template list. Only then issue a certificate.

Returned-certificate validation

Before a certificate is stored, Spiryra checks: the subject CN matches the request, every requested SAN is present, the public key matches the key that generated the CSR, the certificate is not expired, and the signature verifies against your configured trust bundle. The chain is ordered with the issuing CA first.

Housekeeping

  • Rotate connector credentials after vault rotation (11.3) and after any suspicion of exposure.
  • Disable rather than delete a connector that has issuance history; disabled connectors stay in audit records.
  • Every CA-side rejection is surfaced as an HTTP error with the CA message; the inventory is only updated after the CA confirms.

4.2EJBCA REST connector

EJBCA is reached over its REST API with mutual TLS, a bearer token or basic authentication. Spiryra generates the key and PKCS#10 request locally and calls /certificate/pkcs10enroll.

Configuration

FieldTypeNotes
baseUrlURLEJBCA HTTPS origin. Non-HTTPS origins, embedded credentials, paths, queries and fragments are rejected; redirects are disabled.
caNametextIssuing CA. The test proves this CA exists.
certificateProfiletextCertificate profile used for issuance.
endEntityProfiletextEnd-entity profile used for issuance.
authTypeselectmtls, bearer or basic.
username / usernamePrefixtextAPI identity and optional prefix for created end entities.
clientCert, clientKey, trustBundle, token, passwordsecretSealed as one document. The trust bundle should contain the HTTPS server's root and the issuing CA whose chain you want stored.

Required EJBCA role rules

The API identity needs at least: /ca/<issuing-CA>/, /ca_functionality/create_certificate/, /ca_functionality/view_ca/, /ca_functionality/view_certificate/, /ca_functionality/view_certificate_profiles/, /ra_functionality/create_end_entity/, /ra_functionality/view_end_entity/, /ra_functionality/revoke_end_entity/, and the matching create_end_entity, view_end_entity and revoke_end_entity rules for the chosen end-entity profile.

Warning: A missing revoke rule makes EJBCA answer 403; Spiryra reports 502 and leaves the certificate active, because the CA did not confirm the revocation. Fix the rule rather than retrying blindly.

Lifecycle behaviour

  • Test: authenticates to /ejbca/ejbca-rest-api/v1/ca and confirms the configured CA name is present.
  • Issue: generates key and CSR locally, posts to /certificate/pkcs10enroll, then validates the returned certificate.
  • Renew: same path, with the retained key (reuseKey) or a fresh key; the previous certificate becomes replaced.
  • Revoke: parses the real issuer DN and calls PUT /certificate/<issuerDN>/<serial>/revoke?reason=…. Inventory changes only after EJBCA accepts.
  • Templates: discovery reads the Certificate Profiles admin page with the connector's own credential and stores profiles of type End Entity.

Lab verification

cd rust-backend
python3 ../scripts/verify-ejbca-connector-e2e.sh

In the test profile the wizard already generated a paste-ready configuration at deploy/test-ejbca/connector-paste-values.txt. Treat that identity as disposable.

4.3HashiCorp Vault PKI connector

Signs a locally generated CSR through a Vault PKI role, verifies the returned chain, and revokes through the PKI engine. Lab-verified against Vault 1.18.5.

Configuration

FieldTypeNotes
baseUrlURLVault HTTPS origin. Plain HTTP is accepted only for a loopback development server.
mounttextPKI secrets engine mount, default pki.
roletextSigning role. A certificate profile can override it per request.
namespacetextOptional X-Vault-Namespace.
authTypeselecttoken or approle.
tokensecretRequired for token auth.
roleId / secretIdsecretRequired for AppRole auth.
caPemsecretTLS trust anchor for the Vault listener. Not the PKI CA.
pkiCaPemsecretTrust anchor for issued certificates. When empty, the issuing CA returned by Vault is used.

Minimum Vault setup

vault secrets enable pki
vault write pki/root/generate/internal common_name="Lab Issuing CA" ttl=87600h
vault write pki/roles/spiryra allowed_domains="example.com" \
  allow_subdomains=true max_ttl=720h key_type=ec

Grant the Spiryra identity read access to <mount>/ca/pem and <mount>/roles/<role> and update access to <mount>/sign/<role> and <mount>/revoke.

Lifecycle behaviour

  • Test: resolves a token (or performs an AppRole login), reads the CA, and reads the role to prove it exists and report its allowed domains.
  • Issue: POST /v1/<mount>/sign/<role> with the CSR, CN, DNS/IP SANs, requested TTL and format=pem.
  • Renew: same path with a new key by default; reuseKey: true when a retained key exists.
  • Revoke: POST /v1/<mount>/revoke; the serial appears in the Vault CRL.
Note: The role's max_ttl and allowed domains are authoritative. A requested validity can be shortened or refused by Vault — the console reports what the CA actually issued.

4.4ACME v2 connector (HTTP-01, DNS-01, EAB)

Full ACME v2 lifecycle: account registration with a locally generated P-256 account key, order, authorization, finalize, download, chain verification, renewal and revocation.

Configuration

FieldTypeNotes
directoryUrlURLACME directory endpoint. HTTPS; loopback HTTP is accepted only for a local test server.
contactEmailtextOptional account contact.
profiletextOptional CA profile, for example Let's Encrypt tlsserver.
eabKid / eabHmacKeytext / secretExternal Account Binding. Required by some providers; presets do not bypass it.
caPemsecretTLS trust anchor for a private ACME server. Not the issuing CA.
dnsZoneId, dnsZoneName, dnsApiTokentext, text, secretSet all three to enable DNS-01 for every order on this connector. Identifiers outside the zone are refused.
dnsApiBaseURLAdvanced: Cloudflare-compatible API base.
webroottextOptional absolute HTTP-01 challenge directory; defaults to SPIRYRA_ACME_WEBROOT.

Presets are provided for Let's Encrypt staging and production, ZeroSSL, Google Trust Services and SSL.com. A preset fills the directory URL and a known profile only.

HTTP-01

Used when no DNS zone is configured. Spiryra writes <token>.<account-key-thumbprint> to $SPIRYRA_ACME_WEBROOT/.well-known/acme-challenge/<token> and removes it when the authorization completes. nginx serves that path from the same shared volume.

  1. Every requested name must resolve to this Spiryra deployment.
  2. The public proxy must forward port 80 requests for /.well-known/acme-challenge/ to the web container.
  3. Wildcards require DNS-01; a wildcard order on an HTTP-01-only connector is refused with an explicit error.

DNS-01

With a DNS zone configured, Spiryra creates a TXT record for each identifier, waits for propagation, completes the challenge, then removes the record. Cleanup failures are reported rather than hidden, and a journal entry is kept. If the API is killed mid-order, startup recovery removes the orphaned record after restart.

Note: The API container has no egress restriction of its own. Scope the DNS API token to the single zone, and keep the account key sealed — it is never returned by the API, logged or written to audit events.
Verify: In the test profile, python3 scripts/verify-connectors-e2e.py --api-binary target/debug/spiryra-api exercises a full lifecycle against a local ACME stub, including wildcard issuance, a rejected TXT value, a rejected cleanup and an API kill during propagation.

4.5Microsoft AD CS connector

Spiryra talks to AD CS without installing anything on the CA. Two clientless transports are supported, and CA work is never queued to an enrolled agent.

TransportHow it reaches AD CSInstalled on the CA
winrmcertreq/certutil executed on a registered WinRM serverNothing from Spiryra; built-in WinRM only
webAD CS Web Enrollment (/certsrv) over HTTPSNothing from Spiryra; the optional Windows Web Enrollment role

Setup with WinRM

  1. Register the CA host as a WinRM server target with a sealed credential (see 6.6).
  2. Create the connector with transport winrm, select the registered target, the CA name and a certificate template.
  3. Press Test. The response names the CA it reached and lists the templates available on it.
  4. Issue a certificate and confirm it appears in inventory with the CA as issuer.

WinRM and Web Enrollment never receive the CA private key or a CA administrator credential from the console.

Setup with Web Enrollment

  1. Install the Web Enrollment role on the CA and confirm https://<ca>/certsrv answers.
  2. Create the connector with transport web, the Web Enrollment URL, the template and — optionally — basic credentials. Set caPem to trust the site certificate.
  3. Confirm the account is allowed to enroll against the template: a local account on a workgroup CA, a domain account on a domain CA.
  4. Press Test, then issue.

The transport posts the CSR to /certsrv/certfnsh.asp with Mode=newreq, the base64 DER request, CertAttrib=CertificateTemplate:<template>, TargetStoreFlags=0 and SaveCert=yes, then downloads the returned certificate. Absolute and cross-origin certificate links are rejected.

Warning: Web Enrollment cannot revoke — there is no revocation page. Revoke AD CS certificates over WinRM. Both transports refuse to accept a certificate whose CN, SANs, public key or issuing CA do not match the request.
Verify: WinRM issuance was executed end to end against a Windows Server 2025 CA, and Web Enrollment issuance against a domain CA with a complete Web Enrollment role. Both are recorded in CLIENTLESS_ADCS_TRANSPORTS.md and ADCS_WEB_ENROLLMENT_E2E_PROOF.md.

4.6EST connector

RFC 7030 enrolment over HTTPS with a locally generated key and CSR. EST is opt-in per deployment, because the API must be told which hosts it may contact.

Enable EST

  1. Add the EST host to SPIRYRA_EST_ALLOWED_HOSTS in the profile env (comma-separated). An empty value keeps EST unavailable.
  2. Place the EST server's trust anchor in deploy/ca/ — it is mounted read-only at /etc/spiryra/ca — and reference it by absolute path.
  3. Redeploy the profile so the API picks up the new environment.
  4. Create the EST connector with the enrolment URL and trust anchor, then press Test.

Operations

ActionRouteNotes
InspectPOST /api/connectors/est/inspectReads the CA certificates and the enrolment endpoint before creating a connector.
EnrolPOST /api/connectors/est/enrollSends the PKCS#10 request and stores the returned certificate after validation.
Note: EST has no revocation operation in the protocol. Revoke EST-issued certificates at the CA, then reconcile the record in Spiryra. The cacerts, simpleenroll and simplereenroll lifecycle is exercised end to end against a live RFC 7030 EST server with a real issuing CA, and against the local test CA, by scripts/verify-connectors-e2e.py; see ACME_DNS01_EST_WORK_STATUS.md.

4.7SCEP connector

RFC 8894 SCEP enrolment through the packaged CMS exchange, pinned to the CA you configure. SCEP is opt-in, like EST.

Enable SCEP

  1. Add the SCEP host to SPIRYRA_SCEP_ALLOWED_HOSTS (comma-separated).
  2. Place the SCEP CA certificate and any intermediate chain in deploy/ca/.
  3. Redeploy, then create the SCEP connector with the service URL and the challenge password, and press Test.

Operations

ActionRouteNotes
InspectPOST /api/connectors/scep/inspectRetrieves the CA certificate and confirms the endpoint answers.
PKI operationPOST /api/connectors/scep/pki-operationRuns the enrolment exchange for one common name.
Warning: The SCEP implementation is limited to a single common name per request and has no revocation operation. Use another connector when you need SANs or revocation.

4.8CA policy, templates and testing

Template discovery

Where the authority exposes them, Spiryra can pull the certificate profile list and store it for the issue form and policy checks. The result is visible under Connectors and in Permissions → CA policy.

GET  /api/connectors/{id}/templates
POST /api/connectors/{id}/templates/sync
GET  /api/ca/templates

CA policy

CA policy records which templates and profiles a deployment may use, and what key and validity constraints apply. Administrators manage it under Permissions → CA policy:

GET    /api/access/ca-policy
POST   /api/access/ca-policy
PATCH  /api/access/ca-policy/{id}

Testing discipline

  • Test after any credential rotation, CA upgrade or network change.
  • Keep a connector that issues only to a non-production template for smoke tests, then delete that connector when it is no longer needed.
  • Treat a CA-side rejection as an authority policy decision: read the returned message before changing anything in Spiryra.
Note: EJBCA issuance verification stays disabled until SPIRYRA_EJBCA_ISSUED_CA_BUNDLE points at a PEM bundle in deploy/ca/. That bundle is the issuing-CA trust for returned certificates, not the HTTPS trust used to reach EJBCA.

5.1Import certificates

Importing brings an existing certificate and its chain into inventory so expiry tracking, ownership, deployment and reporting cover it.

  1. Open Issue & Import and choose Import.
  2. Paste or upload the PEM certificate. Add the chain when you want a full-chain bundle for deployment.
  3. Fill the operational metadata: owner team, application, business and technical contact.
  4. Save and confirm the parsed subject, SANs, key algorithm, issuer and expiry look correct.
POST /api/certificates/import

What import does and does not do

  • It stores the certificate, chain and metadata; it does not obtain a private key.
  • An imported certificate can be deployed to targets as a file — useful when the key is generated and held elsewhere.
  • Rows that came from discovery or a legacy source may be metadata-only; policy findings only assess fields that were actually recorded.
Verify: The certificate appears in Inventory with status active, the expected issuer CN and a correct days-remaining value.

5.2Issue certificates

Issuance always goes through a configured connector. Spiryra generates the private key locally, builds the PKCS#10 request, and validates the returned certificate before it enters inventory.

Issue from the console

  1. Choose the CA: pick a connector from the issuer list.
  2. Enter the common name and the SANs (DNS and IP). The SAN policy in Settings decides what the console allows.
  3. Choose key algorithm and size, requested validity and whether the key is retained in the vault.
  4. Optionally reference an existing deployment workflow. This screen attaches workflows that already exist in Orchestration and never creates one; the workflow's steps decide which files are written (leaf, chain, full-chain bundle).
  5. Submit and confirm the certificate in inventory.
POST /api/certificates/issue

Issue from your own CSR

If the key must never leave the requesting system, choose I have my own CSR and paste a PKCS#10 request. The CSR is validated (signature, key size, subject and SAN agreement) before it is sent to the CA.

POST /api/certificates/csr/sign
Note: The selected CA profile or role is authoritative for validity and key usage. Where the authority shortens a requested validity, the console records what was actually issued rather than what was requested.
Verify: The certificate is active, the issuer is the connector you chose, every requested SAN is present, and the chain verifies against your trust anchor.

5.3Renew certificates

Renew on demand

  1. Open the certificate and choose Renew.
  2. Keep the same common name, SANs and issuer; decide whether to reuse the retained key or generate a new one.
  3. Submit. The new certificate is stored first; the previous one is then marked replaced.
POST /api/certificates/{id}/renew
POST /api/renewals/{certId}/retry

Scheduled renewal

The renewal scheduler opens work when a certificate enters the configured window, retries failures with backoff, and escalates a repeated-failure streak at most once per streak. Administrators can trigger a sweep:

GET  /api/renewals
POST /api/renewals/run

Renewal plus deployment

A renewal can carry its deployment forward: the referenced workflow runs again — target, destination path, approved service reload and TLS check — and the deployment becomes verified only after the file readback and the presented fingerprint agree.

Verify: Both certificates are visible — the replacement as active and the previous as replaced — and the deployment history shows the new fingerprint.
Tip: Renew well before expiry: expiry windows also drive alerts and the escalation path, so an early renewal keeps the whole delivery chain inside its normal rhythm.

5.4Revoke certificates

  1. Open the certificate and choose Revoke.
  2. Choose the reason. The dialog repeats the subject and serial so you can confirm you are revoking the right certificate.
  3. Confirm. Spiryra asks the CA to revoke and updates inventory only after the CA accepts.
POST /api/certificates/{id}/revoke

What happens next

  • The record is marked revoked with the reason and the CA's confirmation.
  • If a retained key exists for that certificate, it is destroyed.
  • The first revoked result seen by the revocation worker also writes a certificate.ca_revoked audit event and queues one ca_revocation notification.
  • A CA-side revocation also reconciles inventory: the certificate stops reading as valid and is marked revoked once, with a certificate.ca_revocation_reconciled audit event. Only the state changes — the retained key is kept so the revocation can be investigated (an explicit revoke afterwards still destroys it), a certificate that was already replaced keeps that history, and replacing the certificate or rolling back a deployment stays an operator decision.
Warning: Revocation is permanent at the CA. A rejected revoke (for example a missing EJBCA role rule) leaves the certificate active — fix the CA-side permission and retry, rather than assuming it worked.
Note: SCEP, EST and AD CS Web Enrollment have no revocation operation. Revoke those at the CA (over WinRM for AD CS) and reconcile the record in Spiryra.

5.5Retained keys and downloads

Key retention

When retention is enabled at issue or import, the private key is written into the vault under the certificate id, sealed with AES-256-GCM using SPIRYRA_VAULT_KEY. Retention is what makes reuseKey renewal possible.

Downloads

PartRouteControl
CertificateGET /api/certificates/{id}/downloadcertificates.export_public
Chain / bundleGET /api/certificates/{id}/download?part=chaincertificates.export_public
Private keyGET /api/certificates/{id}/download?part=keycertificates.export_private plus an administrator session check in the handler
Warning: Private keys are never deployed to targets. Deployment writes the leaf, the chain and the full-chain bundle only. A key download is an audited, administrator-gated exception for recovery.

Rotation and rewrap

Rotating the vault key rewraps retained leaf keys along with connector credentials and other sealed values (11.3). If a value cannot be opened, nothing is changed and the response names the affected table.

5.6Inventory, owners and expiry

What inventory holds

  • Common name, SANs, key algorithm and size, issuer and issuer kind.
  • Status: active, expiring, expired, revoked or replaced, with days remaining.
  • Ownership: owner team, application, business and technical contact.
  • Source: managed (issued through a connector), imported, or discovered from a target.
  • Linkage to deployments and to orchestration runs that touched it.

Working with the list

  • Filter by status, issuer, owner and expiry window; the expiring view includes already-expired certificates and those inside 90 days.
  • Open a row for the chain, retained-key state, deployment history and audit trail.
  • Assign or change the owner team where you have certificates.manage_owners (administrator-only by design).
  • Export the current view for reporting; expiry and ownership reports page through the full record set.

Scoped visibility

An account with assigned scope only sees certificates it holds a grant for, and never sees another team's rows. Owners always see their own certificates.

Note: A canonical/migration row opens with the detail it kept: when the migration stored the certificate in cert_pem, the console derives SANs, key detail, validity start and PEM from it and the certificate downloads. Event history is not migrated, the canonical schema keeps no intermediate chain, and no private key is ever read. Rows migrated before that column existed still open metadata-only.

5.7Certificate policy settings

Policy settings are what make the console refuse a request that would produce a certificate you cannot manage. Review them before the first issue.

SettingEffect
Expiry warning windowDays before expiry at which a certificate becomes expiring and starts appearing in alerts.
Expiry critical windowDays before expiry at which severity escalates and escalation notifications fire.
Default validityPre-filled lifetime for new requests, subject to the CA's own limits.
Allowed key algorithms and sizesRestricts what the issue form and CSR signing accept.
SAN policyWhich names may appear in a request; wildcards where the CA supports them.
Session lengthConsole session lifetime (sessionHours).
Automatic renewalWhether the scheduler opens renewals inside the window, and the retry policy.
GET   /api/settings
PATCH /api/settings
GET   /api/selftest
Verify: Run the local PKI self-test (Settings → Self-test) after changing policy; it exercises the lifecycle end to end and reports per-check results.

6.1Deployment paths and the target model

There are three ways to deliver a certificate to a server. All three write the same artifact set, record the same evidence and refuse to claim success without reading the file back.

PathTransportBest for
Native agentAgent polls the control planeFleets you manage; no inbound firewall rule; local allowlists decide what may run.
Clientless SSHAPI connects out on TCP 22Linux hosts where installing software is not allowed.
Clientless WinRMAPI connects out on 5985/5986Windows hosts, including AD CS certificate stores.

Objects

  • Server — a registered host: an enrolled agent, an SSH target or a WinRM target, with its sealed credential.
  • Installation (deployment target) — a certificate bound to a server and a destination: absolute path, optional service, optional TLS endpoint/SNI, and a 1–30 second probe timeout.
  • Deployment attempt — one dispatch. It stores the expected SHA-256, queues the job, and only becomes active when the host reads the file back and reports a matching fingerprint.

Register a target

  1. Open Servers and choose the transport.
  2. For an agent, enroll it (6.2) and wait for its first poll. For SSH or WinRM, enter the host, port, user and credential — the credential is sealed immediately.
  3. Confirm the target shows ready: the agent is online, or the SSH/WinRM connection test passed.

Edit a target later

Service, TLS endpoint, SNI and timeout can be changed without detaching the target; the agent, destination, verified certificate and deployment history stay intact.

POST   /api/installations
PATCH  /api/installations/{id}
POST   /api/installations/{id}/deploy
POST   /api/installations/{id}/rollback
Note: Deployment writes public material only — leaf certificate, chain and full-chain bundle. Private keys are never sent to a target, even when a retained key exists in the vault.

6.2Install and enroll the native agent

The agent is a standalone Go executable. It needs no Node.js, npm or JavaScript runtime, and it never listens on a port.

Download

Open Agent Downloads in the console and pick the platform, or fetch directly:

/downloads/agents/spiryra-agent-linux-amd64
/downloads/agents/spiryra-agent-linux-arm64
/downloads/agents/spiryra-agent-windows-amd64.exe

These paths serve allowlisted filenames without authentication, so a host can fetch its own binary before it holds a credential. Checksum or sign the file in your own pipeline if you need supply-chain proof.

Enroll one host

  1. Open Agent Fleet & Enrollment → Enroll agent. The console returns a one-time credential, shown once.
  2. Write it to a restricted file on the host (mode 0600, or an ACL-limited path on Windows).
  3. Set the environment and start the agent.
# Linux
export SPIRYRA_URL='https://spiryra.example.com'
export SPIRYRA_TOKEN_FILE='$HOME/.config/spiryra/agent-token'
export SPIRYRA_RESULT_OUTBOX_DIR='$HOME/.config/spiryra/result-outbox'
export SPIRYRA_ENROLLMENT_TOKEN_FILE='/secure/path/to/bootstrap-token'
chmod +x ./spiryra-agent-linux-amd64
./spiryra-agent-linux-amd64
# Windows (PowerShell)
$env:SPIRYRA_URL='https://spiryra.example.com'
$env:SPIRYRA_TOKEN_FILE="$env:ProgramData\Spiryra\agent-token"
$env:SPIRYRA_RESULT_OUTBOX_DIR="$env:ProgramData\Spiryra\result-outbox"
$env:SPIRYRA_ENROLLMENT_TOKEN_FILE='C:\secure\bootstrap-token'
.\spiryra-agent-windows-amd64.exe

The Windows .exe is the agent runtime; PowerShell only launches or installs it.

Bulk rollout with a campaign token

Instead of a per-host credential, set SPIRYRA_AGENT_ENROLLMENT_TOKEN (at least 32 characters) on the control plane and point each host's SPIRYRA_ENROLLMENT_TOKEN_FILE at that campaign secret. The host self-enrolls and receives its own identity. A campaign token is a shared secret for the duration of the rollout — rotate it afterwards.

A campaign token carries no principal, so on a deployment that hosts more than one tenant set SPIRYRA_AGENT_CAMPAIGN_TENANT to the tenant the campaign belongs to. Every credential it hands out, and therefore every finding those hosts report, is filed there; leave it empty to use the default tenant, which is correct for a single-tenant installation. A value that is not a tenant identifier refuses enrollment instead of falling back, so a typo cannot put a fleet's evidence into an estate that does not own those hosts.

Credential lifecycle

  • The agent persists its credential, commits it, then polls /api/agent/jobs?version=… every 15 seconds.
  • Every SPIRYRA_ROTATE_DAYS (default 30) it prepares a new token, persists it, then commits. Both tokens stay valid until commit finishes, so a crash cannot lock the host out.
  • Removing an agent revokes its credential immediately and clears any pending rotation.

Choose the agent's capabilities

Enrollment asks which job kinds this host may receive. A credential is only handed the jobs its capabilities cover; a queued job for anything else is failed with a denial instead of being delivered later.

CapabilityDelivers
discoveryLocal certificate and Windows store discovery.
deployCertificate file writes, the approved service reload and TLS evidence.
workflowApproved orchestration host steps (copy, command, reload, verify).
windows-pfxPFX import into the machine store and the optional IIS rebind.
tls-monitorMonitored endpoint checks reported back to the console.
upgradeAgent upgrade jobs and upgrade outcome reports.

Change them later from Agent Fleet & Enrollment → Scopes (PATCH /api/agents/{id}/scopes). Narrowing applies to the next poll, and a job that was already dispatched is refused when the capability it needs has been removed. New enrollments start with every capability, so an existing host keeps working until you narrow it.

Verify: The agent appears in the fleet list as online with its build label and its capability pills, and a discovery job completes. An agent that never connects stays never connected rather than silently “enrolled”.
Warning: Without the optional binding below, a token is a bearer secret. It is bounded to this host and to the capabilities it holds, so a stolen credential can only run those job kinds until it is rotated or the agent is revoked. Protect the token file, and revoke the agent if it is lost.

Optional: bind the credential to a client certificate

By default the token is the only thing an agent presents, so anyone who copies the file can poll that host's feed. Bind the credential to one client certificate and the token alone stops working: the request has to arrive through a TLS terminator that verified that exact certificate, and a different certificate is refused. The binding is per agent and optional, so an existing fleet keeps working until you enable it host by host.

  1. Terminate TLS in front of the API with client-certificate verification enabled. The shipped nginx configuration uses ssl_verify_client optional; the API speaks plain HTTP behind it and never sees the certificate itself.
  2. Forward what the terminator verified as x-client-cert-pem (nginx $ssl_client_escaped_cert), x-client-cert-fingerprint (SHA-256 of the DER, what Envoy's %DOWNSTREAM_PEER_FINGERPRINT_256% produces), or both, plus x-client-cert-verify with SUCCESS. The terminator must overwrite those headers; nginx does for the ones it sets.
  3. Bind the certificate when you enroll the host (mtlsFingerprint) or later with PATCH /api/agents/{id}/mtls. Upper case and colon separators are normalised to the stored lower-case digest.
  4. If the API is reachable without going through that terminator, set SPIRYRA_AGENT_MTLS_PROXY_SECRET on the control plane and have the terminator send it as x-spiryra-proxy-secret. The certificate headers are then ignored unless the secret matches, so a direct client cannot forge them.
  5. DELETE /api/agents/{id}/mtls clears the binding, and the same token works again with no certificate.
Verify: With a binding in place, the token alone returns 401 on GET /api/agent/jobs, a different certificate returns 401, and the bound certificate returns 200. Clearing the binding restores token-only access to the same credential. scripts/verify-agent-mtls-e2e.sh runs exactly that sequence against a temporary API.

6.3Agent jobs, readback, reload and TLS verification

A deployment is a small pipeline on the host: write the file, read it back, reload the service, then probe what the socket actually presents. Each stage is recorded.

What the Linux agent needs before it installs anything

Certificate-file installation is gated twice on the host, and both gates are off by default. Until the lab opt-in and a usable recovery key are in place, an install job fails with a named error; the agent does not fall back to writing the file.

RequirementWhat happens without it
SPIRYRA_LAB_REMOTE_CERT_INSTALL=1The explicit opt-in. Unset, every install job fails with remote certificate installation is disabled.
SPIRYRA_INSTALL_ROOTS, SPIRYRA_RECOVERY_JOURNAL_DIR and SPIRYRA_RECOVERY_KEY_FILEAll three are needed together. If any is unset the job fails with agent installation allowlist and recovery configuration required.
A recovery key of exactly 32 raw bytes, kept at mode 0600The contents are checked, not the mode. A missing or short key fails the job with valid 32-byte recovery key unavailable, and batch rollback refuses to run, rather than write a file that cannot be undone.

The recovery key is a durable host secret, not a per-run value: the journal sealed with it before the write must still be readable after a restart. Generate one with openssl rand 32, keep it outside the certificate directories, and back it up separately — a lost key leaves an interrupted deployment with no rollback path. Set SPIRYRA_RECOVER_ON_STARTUP=1 to replay an unfinished journal at boot; the agent then refuses to start, instead of serving jobs, when the key or the journal directory is missing.

The script the agent enforces

  1. Write the certificate material to the destination path, inside SPIRYRA_INSTALL_ROOTS. A path outside those roots is refused.
  2. Read back the file and report its SHA-256. The deployment fails if it differs from the expected fingerprint.
  3. Reload the named service, but only when it appears in SPIRYRA_APPROVED_SERVICES. A service outside the list is not touched.
  4. Probe the TLS endpoint, only if it appears in SPIRYRA_TLS_PROBE_ALLOWLIST; an empty list denies every probe. The requested timeout is capped by SPIRYRA_TLS_PROBE_MAX_TIMEOUT_SECS before any file is written.
  5. Report the leaf certificate actually served — fingerprint, subject and expiry — plus the effective timeout and reload result.

What the control plane requires

  • A timeout and a matching presented fingerprint before a deployment is marked verified.
  • Both the requested and the effective timeout are stored, and audit events carry the effective value.
  • A mismatch fails closed and queues an automatic rollback when the key did not change.
  • Results are delivered from a durable local outbox, so a brief network failure does not lose evidence.

A deployment is a workflow you build and test in Orchestration, then reference from the issue or import screen. Those screens only attach an existing workflow — they never generate one — so the artifact list, reload and verification steps are the same for a first issue and for every renewal.

Verify: The deployment detail shows file=succeeded, the readback digest, reload=succeeded, and a presented fingerprint equal to the certificate's. Only then is the row active.
Note: A reload does not prove the new certificate is in use — only the TLS probe does. Configure the endpoint whenever the service terminates TLS; otherwise the file readback is your only evidence.

6.4Agent fleet and signed upgrades

Fleet view

GET    /api/agents
GET    /api/agents/fleet
POST   /api/agents
POST   /api/agents/bulk
DELETE /api/agents/{id}

The list shows total, online, outdated and never-connected counts, each agent's build label and last poll, and the last 25 jobs. Bulk enrollment creates one identity and one token per host — revoking one agent never disturbs the rest.

How an upgrade works

  1. Publish a release entry in SPIRYRA_AGENT_RELEASE_CATALOG as local JSON: { "releases": [ { "version", "os", "arch", "sha256", "url", "signature" } ] }.
  2. The agent stages a release only when SPIRYRA_AGENT_UPGRADES_ENABLED=1, a pinned SPIRYRA_UPDATE_SIGNING_PUBLIC_KEY exists, and the stage and intent directories are absolute paths.
  3. The Linux package already installs the supervisor, its timer and a root-owned /etc/spiryra-agent/supervisor.env; fill in that file and enable spiryra-upgrade-supervisor.timer. For a hand-managed install use scripts/install-upgrade-supervisor-linux.sh (root) or scripts/install-upgrade-supervisor-windows.ps1 (elevated). Set SPIRYRA_AGENT_EXECUTABLE to the installed agent binary, never to the supervisor.
  4. Create a rollout, dispatch it, and watch outcomes. Agent completion means staged, not installed.
POST /api/agent-upgrades/rollouts
POST /api/agent-upgrades/rollouts/{id}/approve
POST /api/agent-upgrades/rollouts/{id}/dispatch
GET  /api/agent-upgrades/rollouts/{id}/outcomes
POST /api/agent-upgrades/rollouts/{id}/cancel

Installing the agent

scripts/package-agent.sh builds the installable artifacts from one source tree:

  • spiryra-agent_<version>_amd64.deb and _arm64.deb for Debian and Ubuntu.
  • spiryra-agent-<version>-1.x86_64.rpm and .aarch64.rpm for RHEL, Rocky, Alma and SUSE.
  • spiryra-agent-<version>-windows-amd64.zip: the executable, both installer scripts and INSTALL.txt.
  • SHA256SUMS, plus SHA256SUMS.sig when SPIRYRA_PACKAGE_SIGNING_KEY names a key.

A package installs the agent at /usr/local/bin/spiryra-agent, a separate supervisor copy at /usr/local/libexec/spiryra-upgrade-supervisor, the three systemd units, and 0600 root-owned configuration. It does not start anything: the agent has no control-plane URL or credential until you set SPIRYRA_URL and enable spiryra-agent.service.

The gate's agent-packages stage builds these artifacts and installs them in throwaway Debian and EL9 containers, so a package that installs with the wrong file mode, lands a unit where the distribution does not read it, or fails to remove cleanly fails the release.

Warning: These packages carry no distribution signature and no Authenticode signature, and no .msi is produced. Verify SHA256SUMS (and its signature, when the release was signed) before installing; sign the packages with your own key if your fleet policy requires it. An upgrade never installs a package: the supervisor activates a signature-verified executable and rolls back if the health check fails.
Verify: The outcome row moves to staged with a verified signature, the supervisor activates it, and the agent reappears online with the new build label.

6.5Clientless SSH delivery

No agent is installed. The control plane connects with a sealed SSH key, writes the public certificate, reloads an approved service and verifies the file and the TLS endpoint.

Prerequisites on the target

  • SSH reachable from the API container on TCP 22.
  • A user with write access to the certificate path and permission to reload the service.
  • A host key recorded when you register the server — the connection is pinned to it.

Register and deploy

  1. Servers → Add server → SSH. Enter host, port and user, then paste the private key (and an OpenSSH user certificate if your estate requires one).
  2. Save. The credential is sealed with the vault key and never returned.
  3. Create the deployment target: absolute certificate path, optional file mode, optional service reload, optional TLS endpoint.
  4. Deploy. The API writes the file, reloads the approved service, reads the file back and probes the endpoint.

Safety properties

  • The command executed for a reload is fixed by the target configuration, not composed from user input.
  • A failed write leaves the previous file in place; the deployment records the failure instead of retrying silently.
  • Rollback reinstalls the previous certificate with a fresh verification and marks the original deployment restored.
Verify: SSH delivery, verification and rollback were executed end to end on the lab and are recorded in AGENTLESS_DEPLOYMENT_E2E.md.

6.6Clientless WinRM delivery

Windows targets are reached over WinRM with a sealed password. Certificate files are written, services restarted, and the presented certificate reported — all without installing Spiryra on the host.

Prerequisites

  • WinRM enabled. Production should use HTTPS on 5986 with a listener certificate the API trusts.
  • Basic authentication enabled on the listener. Windows disables it by default and refuses every Spiryra request with WinRM authentication failed until it is on. In an elevated PowerShell on the target:
    Set-Item WSMan:\localhost\Service\Auth\Basic -Value $true
    For a lab listener on 5985, unencrypted traffic must be allowed as well: Set-Item WSMan:\localhost\Service\AllowUnencrypted -Value $true and Restart-Service WinRM. Do not enable AllowUnencrypted on a host reachable outside its management network; use 5986 there.
  • An account that can write the destination path and restart the named service.
  • SPIRYRA_AGENTLESS_WINRM_ALLOW_HTTP=true only for a lab using 5985.

Register and deploy

  1. Servers → Add server → WinRM: host, port, user and password (sealed immediately).
  2. Create the deployment target with the Windows path, the service to restart and the TLS endpoint to probe.
  3. Deploy. The API writes the file, restarts the service, then reads the certificate the socket serves and compares its fingerprint.
Note: The probe wraps the served certificate in X509Certificate2 so expiry is read from the actual leaf. Earlier builds read NotAfter from a bare X509Certificate, which produced a null-method error that the evidence parser treated as a failed check.

Script behaviour

ActionSuccessFailure
Write certificateFile digest matchesNon-zero exit; deployment fails
Restart servicereload=succeededUnknown service fails with a legible error
Probe endpointFingerprint matches, expiry recorded, no error lineClosed port or mismatch fails the check
Verify: scripts/spiryra-windows-ca-lab/Verify-SpiryraWinRMDeployScripts.ps1 runs these exact scripts against a running service and a real TLS listener — five checks, zero failures in the recorded container run.
Verify: scripts/verify-winrm-pfx-e2e.py runs the API under WSL against a Windows Server 2025 IIS container and passes every check: a staged import over WS-Management, the HTTP.sys rebind from the placeholder to the imported thumbprint, and a live TLS handshake that presents that certificate.
Verify: The four orchestration steps ran against a live Windows Server 2025 host on 2026-11-02: a real IIS site held the certificate, reload_service restarted W3SVC, and verify matched the fingerprint the site served. Recorded in WINDOWS_LIVE_WINRM_E2E.md; re-run it with scripts/verify-winrm-live-host.py.

6.7Windows PFX store, service-account ACL and IIS

Some Windows services need the certificate in the machine store rather than a file. The agent imports the PFX, binds the private key to the service identity that must read it, and can move — or create — an IIS HTTPS binding for the imported certificate in the same job.

How the import works

  1. Stage the PFX and a DPAPI-protected password file locally on the target. A user-scope blob (ConvertFrom-SecureString) works when the staging account also imports; a machine-scope blob is accepted as well, and is what you need when a service account, WinRM or SYSTEM does the import. The API payload has no password field.
  2. The agent imports into LocalMachine/My with the machine key set; the recorded evidence includes the store, hasPrivateKey and the key method (cng or capi).
  3. Set SPIRYRA_WINDOWS_PFX_SERVICE_ACCOUNT to the SAM or UPN identity that reads the key. A bare local name is machine-qualified before use.
  4. The agent writes an Allow-Read ACE on the key container (Microsoft\Crypto\Keys for CNG, Microsoft\Crypto\RSA\MachineKeys for CAPI), reads it back, and fails the job if it could not be written.

Set SPIRYRA_WINDOWS_PFX_REQUIRE_SERVICE_ACCOUNT=true to fail closed when no identity is configured. The account is host configuration only, so a control-plane caller cannot choose which identity receives private-key access. The control plane also rejects a succeeded result whose ACL evidence is partial.

Verify: In a Windows Server 2025 Server Core container the import reported store=LocalMachine/My, hasPrivateKey=true, method cng, and the key container carried Read, Synchronize for the non-admin account SPIRYRA-SVC. An unresolvable identity fails closed.
Note: The container proves the ACE is written for the right SID, not that the identity can open the key: a Server Core container refuses non-admin logons with 0xC0000142. Confirm effective access on a real or domain-joined host.

IIS certificate changeout

IIS serves TLS through HTTP.sys, which accepts only a certificate that is in LocalMachine\My with its private key. A file deployment of the public PEM therefore cannot renew an IIS binding; the rebind belongs to the PFX import that put the key there.

  1. Open Installations → Windows IIS changeout and choose the channel: a Windows agent that already holds the staged PFX, or a registered WinRM server with no agent.
  2. Name the staged PFX and DPAPI password files and give the certificate's public SHA-256 fingerprint.
  3. Set the IIS site and its HTTPS binding, for example Default Web Site and *:443:www.example.test. Supply both or neither.
  4. The transport imports the certificate, then matches the site's HTTPS bindings: exactly one match is moved to the new thumbprint with the previous one recorded, and a site that has none gets the binding created for this job. The HTTP.sys registration is read back before success is reported.
  5. Optionally name an application pool (agent channel only) to have the job recycle it once the binding is confirmed, so the worker that has to pick up the new key restarts without you. It runs after the bind, outside the rollback path: a failed restart reports the reason and leaves the certificate serving.
  6. The row appears under Windows store imports with the store, private-key ACL, IIS site/binding, requested pool and recycle evidence, previous and applied thumbprints, or the rejection reason.

Agent or WinRM

ChannelWhere the PFX is stagedResult
Spiryra agentUnder SPIRYRA_WINDOWS_PFX_STAGING_DIR on the agent host, which needs SPIRYRA_WINDOWS_PFX_ENABLED=true.Queued (202); the agent reports store, ACL and IIS evidence on its next poll.
WinRM (no agent)Under SPIRYRA_WINRM_PFX_STAGING_DIR on the target. The control plane needs SPIRYRA_AGENTLESS_ENABLED=true and the host in SPIRYRA_AGENTLESS_WINRM_HOSTS.Synchronous; success returns the evidence, a rejection returns 502 and records the remote reason in the history.
Note: The WinRM account must be able to import into the machine store and edit IIS, which needs an administrative identity. Private-key ACL binding is agent-only host configuration; the WinRM request cannot choose an account.

Binding the key to an identity grants use of it, not only an entry in a list. On a live Windows Server 2025 host the agent's import binds NT AUTHORITY\NETWORK SERVICE with the same Set-Acl call it uses in production, and a probe running as that identity signs a payload with the key; an identity without the ACE is refused, and the control leg binds that second identity and has it sign, which shows the refusal came from the grant. Run it with scripts/verify-windows-pfx-identity-e2e.py; it removes the certificate, the probe and any task it created.

A script that does not fit one WS-Man command line is uploaded to the target's temp directory in bounded chunks and run from that file. Sent inline, WS-Man answers The command line is too long and runs nothing, so the control plane never sends the import inline.

POST /api/installations/windows-pfx
{"id":"job_pfx_000001","agentId":"agent_00001",
 "pfxName":"tls-server.pfx","passwordName":"tls-server.dpapi",
 "expectedSha256":"<64 hex digits>",
 "iisSite":"Default Web Site","iisBinding":"*:443:www.example.test"}

# after a confirmed bind, recycle every pool that has to load the new key
{"id":"job_pfx_000002","agentId":"agent_00001",
 "pfxName":"tls-server.pfx","passwordName":"tls-server.dpapi",
 "expectedSha256":"<64 hex digits>",
 "iisSite":"Default Web Site","iisBinding":"*:443:www.example.test",
 "iisAppPools":["DefaultAppPool","Reports Pool"]}

Binding rules and evidence

InputAcceptedRefused
SiteLetters, digits, spaces, ._-; 1–64 charactersQuoting or shell characters, leading or trailing spaces
Bindingip:port:host with * or IPv4, port 1–65535, empty/DNS/leading-wildcard hostIPv6 literals, port zero, unknown scheme shapes
HTTPS bindingExactly one matching HTTPS binding is moved. The agent channel creates one when the site has none; a WinRM import only moves a binding that already existsMore than one match, or a site that does not exist; the job fails without changing IIS
Application poolsUp to eight pool names in iisAppPools (iisAppPool for one), letters, digits, spaces, ._-, 1–64 characters each, with a site and binding; each reused after the bind is confirmedA pool list without a site and binding; an unknown pool; a duplicate name; a pool nobody asked for in the result
Evidenceapplied=true, matching site, binding and thumbprint, created, previousThumbprint (null when the binding was created), and appPools naming each recycled pool (plus appPool for a single one)Missing or partial evidence; a rebind or a recycle reported for a job that did not ask for one; one recycled pool standing in for the rest
Note: The changeout touches one binding: it moves the matching one, or on the agent channel creates the missing one, and never alters another binding on the site. A WinRM import needs that binding to exist first and refuses with the IIS site has no single HTTPS binding matching the requested binding when it does not. A binding the job created is removed again — with its HTTP.sys entry — when the changeout cannot be confirmed, for example because the site is not started, so a failed job leaves no new listener behind. An existing binding is never rolled back; re-importing the previous certificate is the way back. Nothing restarts IIS because HTTP.sys applies the binding immediately; the optional pool recycle restarts each named worker, not the service.
Verify: In a Windows Server 2025 IIS container the wildcard binding moved from its placeholder to the imported thumbprint, and a request for an ambiguous binding failed without changing it. The same container proved creation and rollback: a site with no HTTPS binding was given *:8445:created-pfx.example.test, reported created=true and served the import on a live handshake, while a stopped site's newly created binding and HTTP.sys entry were removed after the job reported it could not confirm them. A leaf issued through an intermediate CA under the lab root was then imported into a real application site with its own app pool: GET / over TLS returned 200 with the application body, the served certificate built a three-level chain (leaf → intermediate → root) where the self-signed leaf reported UntrustedRoot, and the site still answered after Restart-WebAppPool. The job then recycled the pool itself: worker pid 756 was replaced by 2736, the old worker drained and exited, and the site returned 200 with the same certificate, while a job naming a pool that did not exist failed with the certificate still applied. A changeout naming two pools recycled both: SpiryraApp worker 1760 gave way to 832 and SpiryraApp2 worker 2992 to 1724, each reported separately in appPools; when one of two pools did not exist the job failed with that pool marked recycled=false and its reason, the other still recycled, and the certificate stayed applied. A second test moved an SNI binding (*:8444:sni.example.test, sslFlags 1) while a default binding on the same site kept its certificate and no IP-wide binding appeared. The control plane separately rejects partial or unexpected evidence.
Warning: Recycling more than one pool, SNI selection beyond the binding host, IPv6 literals and rolling back an existing binding are not implemented by this job. The WinRM request has no pool field, so a clientless changeout still needs a manual recycle. The agent deletes its staged PFX and password ciphertext after a confirmed import unless SPIRYRA_WINDOWS_PFX_KEEP_STAGED=true; the WinRM channel leaves them on the target for you to remove securely.

6.8Deployment rollback and recovery

Single deployment rollback

  1. Open the certificate's deployment history and choose Rollback on the active deployment.
  2. The previous certificate is reinstalled, read back and verified.
  3. The original deployment is marked restored and the rollback is recorded with its own evidence.
POST /api/installations/{id}/rollback

When a write succeeds but verification fails

If the file matches but the presented fingerprint does not, the deployment fails and — when the key did not change — an automatic rollback is queued. If the key did change, the old certificate cannot restore the service, so the failure is surfaced for an operator instead of pretending to fix it.

Recovery ledger

Actions with no automatic inverse stay visible: run commands and external API calls in a batch rollback are written to a recovery ledger with the command or call to review. The deployment detail lists what was reloaded, what needs manual action and who should review it.

Warning: Rollback restores the previous certificate file. It cannot undo an external side effect — a SaaS upload, a DNS change or a command that already ran. Those entries remain in the ledger until an operator closes them.
Verify: After a rollback, the deployment history shows both attempts, the restored certificate is active, and the file on the host matches the previous fingerprint.

7.1Orchestration concepts

Orchestration turns a repeatable certificate operation into a durable, auditable run: write files, reload services, verify endpoints, wait, ask for approval, call an API.

ObjectMeaning
WorkflowThe definition: ordered steps bound to a certificate and one or more targets.
StepOne action. Host steps run on the target; control-plane steps run in the API.
RunOne execution against one target, with per-step state, output and evidence.
BatchOne run across several targets, created atomically.
ApprovalA gate that stops a run until an administrator decides.

Step kinds

KindRuns whereWhat it does
copy_fileHostWrites public certificate material inside the agent's SPIRYRA_INSTALL_ROOTS.
run_commandHostRuns only a command that exactly matches SPIRYRA_APPROVED_COMMANDS on the agent.
reload_serviceHostReloads or restarts a service in SPIRYRA_APPROVED_SERVICES.
verifyHostProbes an endpoint in SPIRYRA_TLS_PROBE_ALLOWLIST and reports the certificate presented.
discoverHostInventories certificate files and records findings.
windows_pfx_importHost (WinRM only)Imports a PFX the operator staged on the server into LocalMachine\My and can move one IIS HTTPS binding. It calls the same import as the certificate page, so the fingerprint check, the store readback and the history row are identical. Agent and SSH targets are refused: the agent's import runs as a queued job with its own service-account ACL.
api_callControl planeHTTPS request for SaaS uploads and health checks.
send_emailControl planeEmails one or more recipients. Needs SMTP under Settings → Notifications; use Send email notification in the palette.
notifyControl planeQueues email or a configured Slack/Teams/PagerDuty/generic webhook.
delayControl planeBounded wait before the next step.
approval_gateControl planeStops until an administrator approves or denies.

Placeholders

Steps can reference the run's certificate and target: {{cert.pem}}, {{cert.pemEscaped}}, {{cert.chain}}, {{cert.fullchain}}, {{cert.commonName}}, {{cert.serial}}, {{cert.fingerprint}}, {{target.name}}, {{target.host}}, {{target.platform}}, {{target.fileStem}}, {{workflow.name}} and {{run.id}}.

GET  /api/orchestration/meta
GET  /api/workflows
GET  /api/runs
GET  /api/runs/{id}

7.2Build a workflow

  1. Open Orchestration → New workflow and name it for the operation, not the environment.
  2. Bind the certificate and the targets. A target must already be registered (6.1).
  3. Drag steps from the palette onto the canvas and order them. Host steps need an agent, SSH or WinRM transport that the target supports.
  4. Fill each step: destination path and file mode for copy_file; the approved service for reload_service; host, port and SNI for verify; URL, method and body for api_call.
  5. Save, then run against one target first. Batches come after a single run succeeds.

Design rules that keep runs debuggable

  • Keep copy_file → reload_service → verify contiguous. That is the sequence that produces usable evidence.
  • Put an approval_gate before anything that reaches a customer-facing service.
  • Prefer several small workflows to one long chain: a failure then leaves a shorter, clearer state to reason about.
  • Use delay only where a real convergence window exists, not as a substitute for verification.
Verify: The run detail shows each step with its status, duration and output, and the certificate's deployment history links to the run.
Note: An import consumes the staged PFX, so stage it again before a second windows_pfx_import run. If the IIS rebind leg cannot run — no IIS, a missing WebAdministration module, a site that is not there — the step fails and the run reports why from the import history rather than claiming success.
Note: On an SSH or WinRM server, copy_file, run_command, reload_service and verify run clientless through the server's sealed credential — no agent needed. SSH uses a pinned host key and a private key; WinRM uses WS-Management and a sealed password. windows_pfx_import is the WinRM-only exception to “host steps need an agent or a clientless credential”: it is a Windows store write. discover still requires an agent, so a flow containing it is refused when you start it.

7.3Host step safety and conditions

Allowlists decide, not the workflow

Agent variableGovernsDefault
SPIRYRA_LAB_REMOTE_CERT_INSTALLWhether the agent performs certificate-file installation at all.Unset disables every install.
SPIRYRA_INSTALL_ROOTSDirectories a copy_file may write into.Unset denies every write.
SPIRYRA_RECOVERY_KEY_FILEThe 32-byte key that seals the rollback journal.Unset disables installs and batch rollback.
SPIRYRA_APPROVED_COMMANDSExact command strings a run_command may execute.Unset denies every command.
SPIRYRA_APPROVED_SERVICESServices a reload may touch.Unset denies every reload.
SPIRYRA_TLS_PROBE_ALLOWLISTEndpoints a verification may probe.Empty denies every probe.
SPIRYRA_TLS_PROBE_MAX_TIMEOUT_SECSUpper bound on a requested probe timeout.Local cap wins over the request.

A workflow that asks for something outside these lists fails at the host with a clear error. Widening an allowlist is a host-side change, made deliberately on that server — never a control-plane setting.

Allowlists for clientless SSH steps

When a step runs over SSH instead of through an agent, the same model is enforced from the API process, with its own lists so a server does not inherit another path's grants. These are API-host environment variables, and an empty list denies the operation.

API variableGovernsDefault
SPIRYRA_SSH_INSTALL_ROOTSComma-separated directories a clientless copy_file may write into.Unset denies every write.
SPIRYRA_SSH_APPROVED_COMMANDSExact command strings, one per line, a clientless run_command may execute.Unset denies every command.
SPIRYRA_SSH_APPROVED_SERVICESComma-separated services a clientless reload may touch.Unset denies every reload.
SPIRYRA_SSH_TLS_PROBE_ALLOWLISTComma-separated host:port endpoints a clientless verification may probe, resolved on the target.Empty denies every probe.
SPIRYRA_AGENTLESS_SSH_KNOWN_HOSTSThe pinned host-key file. Required: without it no SSH step can open a session.Unset blocks every SSH step.
SPIRYRA_AGENTLESS_SSH_HOSTSOptional host allowlist shared with clientless deployment.Empty means the registered server is the boundary.

A clientless write accepts certificate PEM blocks only, verifies the digest after the move, and records backup: none — there is no sealed journal on the API side, so a failed batch lists the write for operator review instead of claiming it restored the file.

Allowlists for clientless WinRM steps

A server registered with the winrm transport runs the same four host steps from the API process, over the sealed password and the same HTTPS listener a clientless deployment uses. The lists are separate from SSH's, so a Windows fleet never inherits a Linux path grant. An empty list denies the operation.

API variableGovernsDefault
SPIRYRA_AGENTLESS_WINRM_HOSTSComma-separated hosts WinRM may connect to. A server outside it is reported as not runnable.Empty blocks every WinRM step.
SPIRYRA_WINRM_INSTALL_ROOTSComma-separated drive-absolute directories a clientless copy_file may write into, for example C:\ProgramData\Spiryra. Prefix matches respect the directory boundary.Unset denies every write.
SPIRYRA_WINRM_APPROVED_COMMANDSExact PowerShell statements, one per line, a clientless run_command may execute.Unset denies every command.
SPIRYRA_WINRM_APPROVED_SERVICESComma-separated services a clientless reload may restart.Unset denies every reload.
SPIRYRA_WINRM_TLS_PROBE_ALLOWLISTComma-separated host:port endpoints a clientless verification may probe, read from the target.Empty denies every probe.
SPIRYRA_AGENTLESS_WINRM_ALLOW_HTTPPermits port 5985 over plain HTTP. Off by default: 5986 with certificate validation is the supported path.Unset refuses HTTP.

Windows has no service reload, so a reload_service step restarts the service and its evidence records mode: reload next to applied: restart — the run says what happened rather than implying a graceful reload. A WinRM write accepts certificate PEM blocks only, checks the digest before the move and after reading the file back, and also records backup: none.

Conditions

A step can carry a condition over the run's state, so a chain can branch on what actually happened: proceed only when the certificate is active, skip an API call when verification failed, or stop before a reload when the file digest already matches. Conditions are evaluated against recorded step output, not against assumptions.

Warning: Do not use conditions to work around a failed verification. A mismatch means the service is not serving the certificate you wrote — fix that before any dependent step runs.

7.4Approval gates

An approval_gate stops the run before the next step. The run waits in Orchestration → Approvals, where an administrator approves or denies it with a reason.

GET  /api/orchestration/approvals
POST /api/runs/{id}/approval
DecisionEffect
ApproveThe run continues at the step after the gate; the approver, time and reason are recorded.
DenyThe run stops. Steps already completed stay completed; no further step runs.
TimeoutThe gate expires: the run stops waiting rather than proceeding by default.
Note: Approvals are administrator-only. The approvals queue is one of the few surfaces that stays closed even to an account with broad write permissions.
Verify: After approval the run detail names the approver, and a denied gate leaves the following steps untouched.

7.5Parallel groups and multi-target batches

Parallel groups

A workflow may contain bounded, contiguous, read-only groups that run concurrently — for example discovery or verification across several endpoints. Write steps never run in parallel: a reload or a file write stays sequential so the evidence order is unambiguous.

Batches

  • Batch creation is atomic: every target gets a run, or none does.
  • A failed target halts its siblings before their next step, so a broken certificate does not spread across the estate.
  • Approved service reloads inside a batch are re-issued after restores, so the service ends up running against the files that are actually on disk.
  • Batch detail shows per-target state, plus what was restored, what was reloaded and what needs review.
POST /api/runs
GET  /api/runs/{id}
POST /api/workflows/{id}/run-batch
Verify: Batch creation against one invalid target creates no runs at all — the request fails atomically rather than leaving a partial batch.
Tip: Run the batch against a canary target inside the same workflow first; the rollback plan and the recovery ledger are then proven before the wider rollout.

7.6Batch rollback and the recovery ledger

A failed batch compensates every succeeded step, not just file writes. The plan is built in reverse completion order and each part is recorded.

  1. Restore files. Sealed per-step backups are restored first, so the host holds the previous certificate material.
  2. Re-issue reloads. Each approved service reload is issued again so the service runs against the restored files.
  3. Record the irreversible. Steps with no automatic inverse — run_command, api_call and any future kind — are written to the recovery ledger with the kind, reason and the command or call to review.

The batch detail carries reloaded, manual, manualSteps and operatorReview; the audit event stays batch.rolled_back at warning level while listing what an operator must finish.

Warning: The engine cannot reverse the effect of an approved command or external API call. It records them so nothing is silently forgotten — closing them is an operator action.

Restart behaviour

On startup the API resumes queued and running runs, preserves succeeded steps, reattaches host steps to their existing agent job and fails in-flight control-plane steps closed. A database lease prevents two API replicas from driving the same run.

Verify: Kill the API during a run and restart it: the run continues without repeating succeeded steps, and the deployment history still shows one attempt per target.

8.1Server inventory and findings

Discovery finds the certificates already on your servers, so they can be tracked, corrected or promoted into inventory — before they expire in a place nobody is watching.

Run a discovery

  1. Open Servers and choose a ready agent, SSH or WinRM target.
  2. Run discovery. The agent inventories certificate files inside its install roots; SSH and WinRM targets follow the same allowlisted paths.
  3. Review the findings: path, subject, issuer, expiry and the fields that could not be read.
  4. Promote a verified public certificate into inventory, or correct the finding state so it stops appearing as new.
GET   /api/findings
PATCH /api/findings/{id}/state
POST  /api/findings/{id}/import
GET   /api/installations

What discovery does not do

  • It does not read or exfiltrate private keys. Findings are certificate metadata.
  • It does not scan arbitrary paths outside the agent's install roots.
  • It does not merge duplicates silently — a promoted finding becomes a certificate row with its own source.
Note: A finding can be newer than inventory: a certificate renewed by hand on the host appears as a change. Treat that as a signal to bring the host back under management, not as noise.

8.2TLS presentation monitoring

Deployment verification proves what a socket served at one moment. Continuous monitoring proves it kept serving the right certificate afterwards.

Create a monitor

  1. Open Monitoring → Add endpoint and enter host, port and — when the service needs it — SNI.
  2. Choose the expected certificate, or let the monitor record the first observed leaf as its baseline.
  3. Save and run a scan to confirm the endpoint is reachable from the API.
GET  /api/monitor/tls
POST /api/monitor/tls
GET  /api/monitor/tls/{id}/history
POST /api/monitor/scan

What each check records

SignalWhy it matters
Presented fingerprintDetects an unexpected certificate, a rolled-back file or a service that failed to reload.
Subject and SAN coverageDetects a certificate that does not cover the name clients use.
ExpiryDrives alerts before the endpoint starts failing.
Chain and trustDetects a missing intermediate, the most common cause of “works in the browser, fails in the client”.
Probe errorRefused connection, timeout or TLS handshake failure.

Alerts are raised on change and on expiry thresholds, and can be acknowledged per endpoint. History is retained so you can answer “when did this change?” rather than only “is it wrong now?”.

Verify: After a deployment, the monitor's next scan shows the same fingerprint as the deployment evidence. If it does not, the reload did not take effect.

8.3Revocation checks (OCSP and CRL)

For active certificates, the revocation worker polls the authority's OCSP responder or CRL endpoints and records what it finds. This closes the gap between “Spiryra thinks it is active” and “the CA says it was revoked”.

What happens on a revoked result

  1. The certificate's recorded revocation state is updated from the CA answer.
  2. On the first revoked result, a certificate.ca_revoked audit event is written.
  3. One ca_revocation notification is queued to the owner and configured channels — once per certificate, not on every poll.
  4. Inventory state is reconciled to revoked once, with a certificate.ca_revocation_reconciled event. Replacing the certificate and rolling back a deployment remain operator decisions, and the retained key is kept for investigation.

Status and limits

  • A real Google Trust Services CRL was checked in the lab.
  • OCSP runs against a real OpenSSL responder in the test suite (revocation::tests::a_live_responder_reports_good_and_revoked): one certificate it reports good and one revoked, through the same request path the worker uses. The responder is generated by the test, so no external service is needed.
  • A CA that publishes neither OCSP nor CRL cannot be checked — record that as a known gap for that issuer.
Note: Revocation is not a substitute for expiry monitoring. A revoked certificate that is still being served is an incident; a certificate that is merely expiring is a scheduled renewal.

8.4Alerts and acknowledgement

Alerts are derived from certificate expiry and monitoring observations, then acknowledged or resolved by operators. The alert row is the alert's own record, not a copy of the certificate: it keeps the last observation (severity, message, threshold, common name, expiry and days remaining, as seen at the last scan), the alert lifetime (createdAt, lastSeenAt) and the operator status. The list is the live expiry projection merged with those rows, so history survives a certificate leaving the warning window. A scan reopens an alert whose condition returns and reopens an acknowledged alert that becomes critical; a resolved alert stays readable under ?status=resolved.

GET  /api/alerts
POST /api/alerts/{id}/ack
POST /api/alerts/{id}/resolve
SeverityTypical triggerExpected response
CriticalExpired, or inside the critical window, or a revoked certificate still served.Replace the certificate now; confirm the endpoint after deployment.
WarningInside the warning window, or a presented fingerprint changed unexpectedly.Schedule renewal; confirm the change was intended.
InfoDiscovery found a new certificate, or a monitor was created.Decide whether to bring it under management.
Note: Alert persistence is partial by design in this preview: alerts are a read-time projection of expiry plus an acknowledgement row. Acknowledging does not change the certificate; it records that a human has seen it.

9.1Local accounts and roles

Roles

RoleCan readCan writeNotes
ViewerOwn team's certificates and serversNoRead-only; cannot issue, deploy or approve.
OperatorOwn team's resourcesYes, within grants and policyIssues, deploys and runs workflows; cannot change identity or settings.
AdministratorEverythingEverythingIdentity, settings, connectors, approvals, service accounts.

Manage accounts

GET    /api/users
POST   /api/users
PATCH  /api/users/{id}
DELETE /api/users/{id}

Administrators create accounts, change roles, reset passwords, disable and delete. The last active administrator cannot be disabled or deleted — the console refuses rather than locking the deployment out.

Durability

  • Accounts live in clm_users as Argon2id hashes; sessions live in clm_sessions as SHA-256 digests.
  • Sessions survive an API restart until they expire.
  • Disabling an account revokes its sessions immediately.
  • The environment bootstrap administrator exists to create the first real accounts — it is not a shared operational login.
Verify: Create a viewer, sign in as them in a private window, and confirm the issue and deploy actions are refused rather than merely hidden.

9.2Teams, ownership and grants

Roles say what kind of action you may take. Teams, ownership and grants say which resources you may take it on.

Three layers

  • Role — viewer, operator or administrator.
  • Scope — all (every resource) or assigned (only explicit grants). Existing installations default to all, so enabling grants does not silently remove access.
  • Grant — one row binding an account to a certificate or connector with read or manage.

Owners always see and manage their own certificates, even on an assigned account. A write additionally requires an operator-or-higher role and a manage grant.

Configure access

  1. Open Permissions and create or select a team.
  2. Set the account's scope to assigned when it should be limited.
  3. Add grants: choose the certificate or connector and the permission level.
  4. Use Preview to ask what the account would be allowed to do before you save.
GET   /api/access/summary
GET   /api/access/catalog
POST  /api/access/teams
POST  /api/access/owners
GET   /api/access/preview
GET   /api/access/audit

What filtering actually does

  • Inventory, detail, orders and connector reads are filtered by grant; a row is visible when its server or its certificate is granted.
  • Writes outside a grant are refused with 403, not silently dropped.
  • A hidden resource is indistinguishable from a missing one, so grants do not leak existence.
Verify: With an assigned account, a certificate outside the grant is absent from the list and returns 404 on direct navigation.

9.3OpenID Connect

OIDC sign-in uses the authorization-code flow with PKCE, nonce, state and return-path handling, and validates the ID token signature, issuer, audience, expiry and nonce before a session is created.

Configuration

FieldValue
issuerProvider issuer URL. Discovery and JWKS are read from it.
clientIdClient registered with the provider.
clientSecretSecret. Sealed in the vault; never returned by the API.
redirectUriMust be <public origin>/api/auth/oidc/callback and registered exactly at the provider.
scopesUsually openid profile email.
labelButton text on the sign-in page.
defaultRoleRole for a first-time federated account.
tokenAuthbasic (client secret basic) or post (client secret post).

Set it up

  1. Register the application at the provider with the exact redirect URI above.
  2. In Settings → Identity providers, enter the issuer and client values and save.
  3. Use the built-in Check action: discovery, JWKS reachability, issuer and client consistency are reported per check.
  4. Sign in through the provider in a private window and confirm the resulting role and team.
GET  /api/auth/providers
GET  /api/auth/oidc/login
GET  /api/auth/oidc/callback
Note: Federated accounts have no local password. Keep at least one local administrator so an identity-provider outage does not lock the deployment out.

Signing in once a provider is configured

The provider becomes the default. Opening the console signed out redirects to it after a four-second hold, so somebody who only knows the site address never chooses a method. The hold is deliberate: the same page carries the escape hatch, and a redirect nobody can interrupt is a redirect that locks you out when the identity provider is down.

  • Bypass SSO on the sign-in page stops the redirect and opens the local sign-in address.
  • /login/local reaches the same screen directly, with no redirect. This is the address to bookmark and to keep in a runbook. GET /api/auth/providers returns it as localLoginUrl, and the console reads that value instead of hardcoding a path.
  • Local (break-glass) accounts sign in with username and password, exactly as they did before federation was configured. Password expiry, the administrator recovery token and the expired-password change all work there.
  • API key signs in as a scoped service account. Paste the token issued on Settings → Service Accounts; the console presents it as a bearer credential, so the session shows only the resource types and actions that token was scoped for, and never becomes an administrator session. Revoking the token ends the session as soon as the next request is made, and the key is held in the tab's session storage rather than a cookie.
Verify: With OIDC or SAML configured, an ordinary visit redirects to the provider; /login/local does not redirect and signs in a local administrator; an API key reaches the console with only its scoped pages and is refused after revocation; and signing out of an API-key session leaves no cookie behind.

9.4SAML 2.0

SAML sign-in uses SP-initiated AuthnRequests with signed assertions, replay protection and strict issuer matching. Once it is configured it is the default sign-in method; the break-glass address is /login/local (see 9.3).

Configuration

FieldValue
entryPointIdentity provider SSO URL (HTTP-Redirect binding) from its metadata.
idpIssuerIdP entity ID. Assertions from any other issuer are rejected.
idpCertSecret. IdP signing certificate as a PEM block or the base64 value from metadata.
spEntityIdService provider entity ID. Leave blank to use the metadata URL.
acsUrlAssertion consumer service: <public origin>/api/auth/saml/acs.
defaultRoleRole for a first-time federated account.
allowIdpInitiatedWhether unsolicited assertions are accepted. Off is safer.

Exchange metadata

  1. Download /api/auth/saml/metadata and give it to the IdP administrator, or copy the entity ID and ACS URL.
  2. Map the IdP's user identifier to the account email used in Spiryra.
  3. Paste the IdP certificate and SSO URL here, then save.
  4. Sign in through the IdP in a private window and confirm the role.
GET  /api/auth/saml/metadata
GET  /api/auth/saml/login
POST /api/auth/saml/acs
Verify: A replayed assertion is refused, and an assertion from a different issuer is refused — both are covered by the SAML test suite.

9.5SCIM 2.0 provisioning

SCIM lets your identity provider create, update, deactivate and group users automatically. Spiryra exposes the standard Users and Groups resources plus discovery documents.

Endpoints

GET    /scim/v2/ServiceProviderConfig
GET    /scim/v2/ResourceTypes
GET    /scim/v2/Users          POST /scim/v2/Users
GET    /scim/v2/Users/{id}     PUT  /scim/v2/Users/{id}
PATCH  /scim/v2/Users/{id}     DELETE /scim/v2/Users/{id}
GET    /scim/v2/Groups         POST /scim/v2/Groups
GET    /scim/v2/Groups/{id}    PUT  /scim/v2/Groups/{id}
PATCH  /scim/v2/Groups/{id}    DELETE /scim/v2/Groups/{id}

Connect the provider

  1. In Settings → Identity providers → SCIM, generate a SCIM token. It is shown once.
  2. Configure the provider with the base URL https://spiryra.example.com/scim/v2 and that token.
  3. Map IdP groups to Spiryra roles or teams under Permissions.
  4. Provision one test user and confirm they appear with the expected role before enabling the full directory sync.

Fail-closed behaviour

  • A SCIM-managed account is always evaluated against custom policies; its legacy scope never widens access.
  • Removing the last group-to-role mapping narrows the account instead of leaving it with its previous role.
  • Deactivating a user removes access immediately; deleting a user preserves the audit trail.
Warning: Do not delete the group mappings while a sync is running. Preview an affected account (GET /api/access/preview) before and after the change.

9.6API service accounts

Machine credentials for automation that should not sign in as a person. A token carries an explicit allow-list drawn from the console's permission catalogue. It can be a bearer secret, or a certificate credential with no secret at all.

Credential shape

Authorization: Bearer csa.<token id>.<secret>

The secret is 32 random bytes, base64url, 43 characters. Only its SHA-256 digest is stored, the plaintext is returned once at creation, and listing never returns the digest or the credential.

Certificate credential (no secret)

Issue a token with clientCertificate set to the PEM of the certificate the automation host will present, and no secret is created or returned: the response carries authMethod: "mtls" and the bound fingerprint instead. From then on the API accepts that certificate — forwarded by the TLS terminator in x-client-cert-fingerprint (or x-client-cert-pem) with x-client-cert-verify: SUCCESS — as the credential. A stolen bearer token cannot exist because there is none, and one certificate backs one active credential.

curl -sS -X POST https://spiryra.example.com/api/service-accounts/svc_…/tokens \
  -H "Authorization: Bearer $SPIRYRA_TOKEN" -H "Content-Type: application/json" \
  -d "{\"name\":\"api client\",\"scopes\":[\"certificates.read\"],\"clientCertificate\":\"$(cat client.pem)\"}"

The terminator must overwrite those headers; set SPIRYRA_AGENT_MTLS_PROXY_SECRET to require a shared proxy secret when it cannot.

Create an account and token

curl -sS -X POST https://spiryra.example.com/api/service-accounts \
  -H "Authorization: Bearer $SPIRYRA_TOKEN" -H "Content-Type: application/json" \
  -d '{"name":"inventory sync","description":"nightly report"}'

curl -sS -X POST https://spiryra.example.com/api/service-accounts/svc_…/tokens \
  -H "Authorization: Bearer $SPIRYRA_TOKEN" -H "Content-Type: application/json" \
  -d '{"name":"report","scopes":["certificates.read","servers.read"],"expiresInDays":90}'
RoutePurpose
GET /api/service-accountsAccounts and token metadata; never secrets.
POST /api/service-accountsCreate an account, optionally bound to an owner team with teamId.
PATCH /api/service-accounts/{id}Change disabled and/or teamId (send an empty string to clear the team); disabling also revokes every live token.
POST /api/service-accounts/{id}/tokensIssue a scoped token, optionally with an expiry.
DELETE /api/service-accounts/{id}/tokens/{token_id}Revoke one token immediately.

How a token is authorized

  1. Administrator-only paths and methods are refused outright.
  2. The route is mapped to the action it performs; an unclassified route is refused.
  3. That action must appear in the token's scopes. Scopes match exactly — certificates.read cannot issue, renew or revoke.

Team-scoped accounts

An account without a team is instance-wide: a certificates.read token sees every certificate, which suits inventory export and reporting. Bind the account to an owner team instead and the same token reaches only the certificates, servers, workflows, connectors and agents that team owns — the action scope still applies, but the object scope is the team. A certificate owned by another team answers 404, exactly as it does for a scoped person. The team must already exist under Roles & Permissions; a misspelt id is refused rather than quietly narrowing the account to nothing.

Verify: With a read-only token, GET /api/certificates returns 200 and POST /api/certificates/{id}/revoke returns 403. Revoking the token makes the next call 401. With a team-bound token, the list shows only that team's certificates and another team's id returns 404.

Manage tokens in the console

Settings → Service Accounts (administrator only) manages the same API: an account list with its team scope, live-token counts and a Disable action that revokes every live token, a token list with scope summary, authentication method, expiry, last use and state, and an issue dialog. The dialog takes a name and lifetime, optionally binds a client certificate instead of issuing a secret, and shows scopes as checkboxes grouped by resource, built from the same permission catalogue the Roles page uses. Actions whose handlers require an administrator session are shown disabled with the reason, because a token can never reach those surfaces. A bearer plaintext appears once after issue, with a copy button; a certificate credential shows its fingerprint and no secret.

Full detail, including the credential format and the test coverage, is in SERVICE_ACCOUNTS.md.

9.7The administrator boundary

A set of surfaces is administrator-only no matter what grants or scopes an account holds. The boundary is explicit, method-scoped and audited by a static check.

Administrator-only prefixes

/api/access/     /api/users        /api/settings     /api/notifications
/api/selftest    /api/agent-upgrades/                /api/agentless/
/api/connectors/scep/   /api/connectors/est/          /api/connectors/adcs/
/api/demo/       /api/enterprise/deployment/          /api/migration/
/api/service-accounts

Plus exact method/path pairs such as POST /api/renewals/run, POST /api/monitor/scan and GET /api/orchestration/approvals, and suffix rules such as /api/runs/{id}/approval.

Deliberate exceptions

  • GET /api/notifications — an owner reads their own delivery history.
  • GET /api/agent-upgrades/rollouts — read-only rollout status.
  • GET /api/enterprise/deployment/recovery — read-only recovery state.
  • POST /api/agentless/certificates/execute — scoped clientless delivery.
  • Migration reconcile routes — method-scoped so a sibling verb cannot inherit the exception.

Why it is trustworthy

  • Exceptions are listed per HTTP method, so adding a method to the same path does not widen access.
  • An unclassified API route is refused for non-administrators instead of falling through.
  • scripts/audit-rust-authorization.py fails the release gate if a route is not classified. It currently reports 219 route-method pairs with 0 needing classification.
Verify: As an operator, a request to any prefix above returns 403 — including the ones whose feature you can otherwise use, such as notifications settings.

10.1SMTP email

Configure the relay

  1. Open Settings → Notifications (or the Notifications page) and select SMTP.
  2. Enter host, port, TLS mode, sender address and the public console URL used in links.
  3. Enter the username and password. The password is sealed and never returned.
  4. Send a test message to an address you control.
SettingNotes
Host and portUse 587 with STARTTLS, or 465 with implicit TLS, unless your relay is internal.
SenderMust be an address your relay accepts; it appears in From.
Public URLUsed for links in recovery and alert mail; must match SPIRYRA_PUBLIC_ORIGIN.
CredentialsStored sealed; rotate after vault rotation.

Console settings take precedence over environment variables, so the relay can be changed without a restart. Password recovery (3.3) uses the same saved relay.

GET   /api/notifications/settings
PATCH /api/notifications/settings
POST  /api/notifications/test
Verify: The test message arrives, and its link opens the console at the configured public origin.

10.2Webhook channels

Channels deliver the same events as email: expiry warnings, deployment failures, revocation notices and escalation. Slack, Teams, PagerDuty v2 and a generic JSON channel are supported.

Add a channel

  1. Open Notifications → Channels → Add and choose the type.
  2. Paste the webhook URL or routing key. Secrets are sealed and never returned.
  3. Choose which events the channel receives.
  4. Send a test and confirm the message in the target system.
GET    /api/notifications/channels
POST   /api/notifications/channels
POST   /api/notifications/channels/{id}/test
DELETE /api/notifications/channels/{id}

Transport rules

  • HTTPS only; redirects are refused.
  • Private and loopback targets are blocked unless explicitly enabled.
  • Delivery retries are recorded in the same history as email, with the last error preserved.
Warning: A webhook URL is a credential. Rotate it at the provider if the channel is copied between environments, and do not enable private targets unless you accept that the API can reach internal services.

10.3Escalation and delivery history

Escalation

When a renewal or deployment keeps failing, escalation ensures a human is told. Configure a failure threshold and a fallback recipient list; when the threshold is crossed, the owner and fallback recipients are notified.

  • Automatic renewal opens at most one escalation per failed renewal streak.
  • A deployment episode escalates once, not on every retry.
  • Revocation notice from the CA is a one-time event per certificate (8.3).

Delivery history

GET  /api/notifications
POST /api/notifications/{id}/retry

Each entry records the channel, event, attempt count, outcome and last error. A failed delivery can be retried without re-running the triggering operation.

Verify: Break a webhook URL deliberately, trigger a test, confirm the failure appears in history, fix the URL and retry — the second attempt should succeed from the same entry.
Note: Notifications are best-effort delivery, not a queue with guaranteed ordering. For a hard guarantee, point a webhook at your own durable receiver.

11.1Health and readiness

EndpointQuestion it answersTypical output
/healthzIs the process serving?ok
/readyzCan it do its job?200 with per-check status, or 503 when a check fails.
/api/metaWhat is this deployment?Profile, available connector kinds, demoAvailable, productionReady: false.
/metricsWhat is it doing?Prometheus text: request counters, queue states and agent liveness. Not proxied by the web container.

Readiness checks

  • data-volume — the data directory is writable and present.
  • database — SQLite opens and passes quick_check.
  • schema — the database version is not newer than this binary. An older database is upgraded in place; a newer one reports 503 with the found and supported versions, because an old API must not read a row shape it does not know.
  • identity — the account schema and password policy load. A failed migration reports 503 instead of leaving sign-in broken behind a healthy process.
  • connectors — at least one CA connector is configured and reachable; a fresh install is degraded here, which is expected.

A fresh install therefore answers 200 degraded until a CA is connected. A failure in any check returns 503 — that is the signal to stop traffic, not to restart blindly.

Verify: curl -fsS https://spiryra.example.com/healthz and curl -i https://spiryra.example.com/readyz. The startup log prints the same summary plus each check's detail.

Metrics

GET /metrics answers in Prometheus text format from the API process. The web container does not proxy it, so it is reachable from the container network and from whatever scrapes the API directly. Set SPIRYRA_METRICS_TOKEN to require Authorization: Bearer <token>; the production profile requires it, and without a token the endpoint answers anyone who can reach the API port.

SeriesTypeMeaning
spiryra_http_requests_totalcounterResponses by method, normalised route and status. Ids collapse to :id, so the label set stays bounded.
spiryra_agentsgaugeEnrolled, unrevoked agents by online/offline.
spiryra_fleet_jobsgaugeFleet queue rows by state, excluding Windows PFX imports.
spiryra_windows_pfx_importsgaugeWindows PFX imports by state, including failed ones.
spiryra_uptime_seconds, spiryra_infogaugeProcess age and build identity.
Note: Alert on spiryra_http_requests_total with status="429" rising, on spiryra_windows_pfx_imports{state="failed"} and on /readyz leaving 200.

11.2Backup and restore

Back up a running profile

./scripts/backup-profile.sh test
./scripts/backup-profile.sh production

The script uses SQLite's online backup API for every *.sqlite file, verifies each snapshot with PRAGMA integrity_check, copies the remaining data-volume files, and writes a checksummed manifest.json. Default destination:

backups/<profile>-<UTC timestamp>/

Restore

./scripts/restore-profile.sh test backups/test-20261029T010203Z --yes
  1. Verify the manifest and every checksum.
  2. Take a fresh pre-restore backup automatically.
  3. Stop the API, replace the database files, restore ownership.
  4. Start the API and wait for /healthz.

Rehearse the production path

./scripts/verify-production-deploy.sh --check
./scripts/verify-production-deploy.sh --backup-only
./scripts/verify-production-deploy.sh --restore-drill --yes

The drill takes a backup, verifies checksums, restores the same archive, checks health and compares certificate and account counts. It causes a short outage and keeps the pre-restore backup.

Warning: Back up SPIRYRA_VAULT_KEY separately. The archive may include a generated keyring but never an environment vault key. Without the wrapping key, sealed connector credentials and retained leaf keys are unreadable — a backup you cannot use.
Verify: Restore into a scratch profile and sign in. A restore that boots but cannot open connector credentials is not a successful restore.

11.3Vault key rotation

Changing SPIRYRA_VAULT_KEY on its own makes every sealed value unreadable. Rotate in three steps instead.

  1. Introduce the new key alongside the old. Set the new value in SPIRYRA_VAULT_KEY and put the old value in SPIRYRA_VAULT_KEY_PREVIOUS (comma-separate several), then redeploy. Reads and new writes keep working.
  2. Re-wrap. As an administrator open Settings → General → Vault key rotation and confirm Re-wrap sealed values. One transaction decrypts and re-encrypts retained leaf keys, connector credentials, registered-server SSH keys, WinRM passwords, notification secrets, the SMTP password and the OIDC client secret. If any value cannot be opened, nothing changes and the response names the affected table.
  3. Retire the old key. Remove SPIRYRA_VAULT_KEY_PREVIOUS, redeploy, and back up the new key separately.
Verify: After step 3, open a connector test, download a retained key as an administrator, and send a notification test. All three must work with only the new key present.
Note: A recorded live rotation rewrapped six sealed values with zero skips, and every connector still opened them after the previous key was removed.

11.4Background workers and leases

Renewal scheduling, revocation checks, TLS monitoring, deployment dispatch and orchestration all run inside the API process. Work is claimed through database leases so two replicas do not do the same job twice.

  • Orchestration runs hold a lease while a replica drives them; a second replica cannot pick up the same run.
  • Two API replicas were exercised and split background work into distinct leases.
  • On startup a replica resumes queued and running runs, reattaches host steps to their existing agent jobs and fails in-flight control-plane steps closed.
  • An offline agent leaves its deployment queued — there is no automatic replay, so a queued job is a visible state, not a silent retry loop.
Note: Leases are per job, not a global singleton. Running one replica is simpler to reason about; running two is supported but every background action must stay idempotent.

11.5Audit log and events

Every security-relevant action is written to an append-only audit log with actor, action, severity, summary and structured detail. The console reads it as an event stream.

GET /api/events
GET /api/access/audit

Representative actions

ActionSeverityWhen
certificate.issuedinfoA CA returned a validated certificate.
certificate.revokedwarningThe CA confirmed a revocation.
certificate.ca_revokedwarningA CA-side revocation was detected by the worker.
service_token.createdwarningA service account token was issued.
service_token.revokedwarningA token was revoked.
batch.rolled_backwarningA batch rolled back, with the manual actions listed.
notification.settings_updatedinfoThe delivery configuration changed.

What audit never stores

  • Private keys, tokens, passwords or sealed values.
  • ACME account keys and DNS API tokens.
  • Full certificate bodies beyond what the record needs.
Verify: Issue a certificate and open Activity. The event names the actor, the certificate and the connector — and contains no secret material.

11.6Logging and diagnostics

Container logs

docker compose -p spiryra-production logs -f api
docker compose -p spiryra-production logs -f web

On startup the API prints the readiness summary and each check's detail. Connector and deployment errors appear with the CA or host message included.

What to collect before asking for help

  1. The output of /readyz and /api/meta.
  2. The failing request's status code and the JSON body — errors carry a message, and authorization failures carry a code.
  3. The relevant container log lines, with secrets redacted.
  4. For deployment failures: the deployment detail, including reload result and presented fingerprint.
  5. For connector failures: the connector Test output, not just the issue error.
Warning: Logs can contain hostnames, usernames and certificate subjects. Redact before sending them outside your organization, and never include an env file, vault key or bearer token.

12.1Security model

PropertyHow it is enforced
No internal CA in productionThe release build has no in-process CA; production refuses a local-ca kind or demo seed. The test CA exists only in the test profile.
No private keys on targetsDeployment writes leaf, chain and full-chain bundle only. Retained keys require an administrator-gated download.
Secrets at restAES-256-GCM with SPIRYRA_VAULT_KEY; connector secrets are bound to the connector id; the API returns field names, never values.
Session integritySHA-256 digests in clm_sessions; CSRF token bound to the session; same-origin check on browser writes.
Authorization depthRole check, resource grants, route classification gate, and handler-level ownership checks — each independently enforced.
Host-side limitsAgent install roots, approved commands, approved services and TLS probe allowlists. The control plane cannot widen them.
Outbound TLSVerification is mandatory; redirects refused; embedded URL credentials rejected; no insecure-skip switch.
AuditabilityAppend-only events for issuance, revocation, tokens, approvals, rollbacks and settings changes.

Trust boundaries

  • The Docker host is trusted: it holds the env file and the vault key.
  • The CA is authoritative for policy and validity; Spiryra validates what comes back but does not overrule it.
  • The target host is semi-trusted: it enforces its own allowlists and reports what it did.
  • The browser is untrusted: every write is authorized server-side, and tokens never reach JavaScript beyond the CSRF value.

12.2Hardening checklist

  1. Use the production profile with an exact HTTPS public origin and secure cookies.
  2. Keep port 8080 on loopback; only the TLS proxy is reachable.
  3. Lock the env file to mode 0600 and store the vault key in your secret manager separately from backups.
  4. Replace the bootstrap administrator with named accounts; keep one local administrator for IdP outages.
  5. Use HTTPS for WinRM (5986) and keep SPIRYRA_AGENTLESS_WINRM_ALLOW_HTTP unset.
  6. Set agent allowlists explicitly and leave them empty where the capability is not needed.
  7. Scope tokens narrowly: service accounts get read scopes unless they must write, plus an expiry.
  8. Turn on notifications and escalation so expiry and failures reach a person.
  9. Rehearse restore and vault rotation before you need them.
  10. Run the release gate (14.3) before each production change window.
Warning: No security review of this preview has been completed. The production gate refuses to pass without a separately signed independent review for exactly this reason.

12.3Known limits

AreaLimit
Agent transportDevice attestation is not implemented. Credentials are scoped per agent (see 6.2) and can be bound to a client certificate, which makes a stolen token useless without that certificate; an unbound credential is still a bearer secret, so a stolen token acts within its own host's capabilities until it is rotated or revoked.
Agent packagingDEB and RPM packages are built by scripts/package-agent.sh, carry the agent, a separate supervisor and systemd units, and are installed and removed in throwaway Debian and EL9 containers by the gate (agent-packages). They are not signed by a distribution or with Authenticode, so integrity rests on SHA256SUMS and its detached signature when a release key is configured. Windows ships an executable archive with the scheduled-task installer, not an MSI. Upgrades still activate a verified executable rather than running the package manager.
EnrollmentSelf-enrollment is rate limited per client address (30/hour by default) and sign-in at 10 per 5 minutes. The windows live in the API database, so replicas that share it share one budget. The client address is the socket peer unless the peer is listed in SPIRYRA_TRUSTED_PROXIES, which is what makes a proxy's X-Real-IP believable; keep a proxy limit in front of enrollment as well.
Fleet listingAgents and the recent-job list beside them both page (limit/offset and jobLimit/jobOffset; agents default 500, max 1000; jobs default 25, max 200), each with its own nextOffset cursor. The console shows both totals, so older jobs are reachable.
SCEPOne common name per request, no revocation.
ESTNo revocation operation in the protocol.
AD CS Web EnrollmentIssuance only — no revocation.
EJBCA approvalsAn approval-gated enrollment is recorded instead of lost: Spiryra keeps the EJBCA approval request id (and the sealed key when it generated one) and polls POST /certificate/{id}/finalize until an RA approves it, then records the certificate through the normal inventory path. Polling stops after SPIRYRA_EJBCA_APPROVAL_MAX_POLLS failed attempts and the row is marked expired. The collector, the real finalize call, recorded errors and expiry ran against a live EJBCA 9.6 CE CA; an approval-gated certificate profile is an EJBCA admin-UI setting, so the approve-then-issue leg is covered by tests rather than a live RA approval.
AlertsAn alert row now keeps its own lifetime: first and last sighting, the last observed expiry data, and its status. A scan refreshes the observation, reopens an alert whose condition returns, and reopens an acknowledged alert that escalates from warning to critical; resolving an alert whose certificate has left the warning window keeps it readable under the status filters. The list still merges the live expiry projection with those rows, so an alert appears as soon as its certificate enters the window, without waiting for a scan.
Canonical recordsThe list stays a metadata projection. Opening one migrated row reads its stored cert_pem (the public certificate) and derives SANs, key detail, validity start and PEM from it; the certificate download serves it. Event history is not migrated, there is no intermediate chain in the canonical schema, and a database that predates cert_pem still opens metadata-only. No private key column is ever read.
Service accountsAn account can be bound to one owner team, so its token reaches only that team's certificates, servers, workflows, connectors and agents; unbound accounts stay instance-wide. A token can also be a certificate credential with no secret, resolved from the client certificate the TLS terminator reports — the same header trust model as agent mTLS, so the terminator must overwrite those headers (or a shared proxy secret must be configured). One certificate backs one active credential.
OCSPThe worker's request path is exercised against a real OpenSSL responder (good and revoked certificates) in cargo test, so a broken request, a nonce the responder rejects or a status the parser mishandles fails the build. The responder is generated by the test; it is not your issuer's, and the scheduled refresh still needs an operator run against a production CA.
WinRM deliveryFile install, restart and presentation are verified as scripts, and a staged PFX import is verified end to end from the API against a Windows container (scripts/verify-winrm-pfx-e2e.py). The four orchestration steps are also proven against a live Windows Server 2025 host (scripts/verify-winrm-live-host.py): store import, HTTP.sys rebind, live handshake, a real W3SVC restart and a probe of the served certificate. Private-key ACL binding as a service identity remains agent-side: scripts/verify-windows-pfx-identity-e2e.py runs the agent's own import on a live Windows Server 2025 host and then signs with the bound identity, so the binding is proven to grant real use of the key, not just the right ACE.
Windows PFXEffective access is proven, not just the ACE. On a live Windows Server 2025 host the agent's own import binds the key to NT AUTHORITY\NETWORK SERVICE; a probe running as that identity signs with the key (a real 2048-bit signature), NT AUTHORITY\LOCAL SERVICE is refused, and the control leg binds that second identity and has it sign, so the refusal is the missing ACE rather than an identity that cannot use CNG keys. The negative leg reports the refusal as Invalid provider type specified rather than a plain access-denied message, so the assertion is that no signature is produced. Still unproven: a domain service account, and IIS serving with a key ACL'd to a non-administrator pool identity.
Note: Each limit above is a conscious gap with evidence, not an unknown. NATIVE_RELEASE_STATUS.md records how each was tested and what would close it.

13.1API authentication and CSRF

Credential types

CredentialHeaderUse
Console sessionCookie: spiryra_sessionBrowser console; writes also need the CSRF header.
Migration admin tokenAuthorization: Bearer …Bootstrap, scripts, recovery. Administrator identity.
Agent tokenAuthorization: Bearer …This host's job feed and results only, limited to the capabilities its credential holds (6.2).
Service account tokenAuthorization: Bearer csa.<id>.<secret>Scoped automation (9.6).

Sign in

POST /api/auth/login
{ "username": "admin", "password": "…" }

200 OK
Set-Cookie: spiryra_session=…; HttpOnly; SameSite=Strict
Set-Cookie: spiryra_csrf=…; SameSite=Strict
{ "user": { "username": "admin", "role": "admin" }, "csrfToken": "…" }

Write requests from a browser

POST /api/certificates/issue
Cookie: spiryra_session=…
X-CSRF-Token: <value from the csrfToken field or the spiryra_csrf cookie>
  • A missing or wrong CSRF token returns 403.
  • The request Origin must match the request Host; a mismatch returns 403.
  • Bearer credentials — migration tokens, agents and service accounts — are not subject to the browser CSRF rule.
POST /api/auth/logout
GET  /api/auth/me
GET  /api/auth/providers
Note: The CSRF cookie is scoped to /api and is not HttpOnly by design, so the console can read it.

13.2Endpoint reference

Representative routes grouped by capability. Every console screen uses these; the authorization gate classifies each route-method pair before a handler runs.

Certificates

GET    /api/certificates
GET    /api/certificates/{id}
POST   /api/certificates/issue
POST   /api/certificates/csr/sign
POST   /api/certificates/import
POST   /api/certificates/{id}/renew
POST   /api/certificates/{id}/revoke
GET    /api/certificates/{id}/download[?part=chain|key]
GET    /api/certificates/{id}/installations
GET    /api/certificates/{id}/automation

Connectors and CA policy

GET    /api/connectors
GET    /api/connectors/specs
POST   /api/connectors
PATCH  /api/connectors/{id}
POST   /api/connectors/{id}/test
GET    /api/connectors/{id}/templates
POST   /api/connectors/{id}/templates/sync
POST   /api/connectors/est/inspect      POST /api/connectors/est/enroll
POST   /api/connectors/scep/inspect     POST /api/connectors/scep/pki-operation
GET    /api/ca/templates

Deployment and fleet

GET    /api/installations               POST /api/installations
PATCH  /api/installations/{id}
POST   /api/installations/{id}/deploy
POST   /api/installations/{id}/rollback
GET    /api/agents                      POST /api/agents
POST   /api/agents/bulk                 GET  /api/agents/fleet
DELETE /api/agents/{id}
GET    /api/agent/jobs                  POST /api/agent/jobs/{id}/result
POST   /api/agent-bootstrap/enroll
POST   /api/agent/token/prepare         POST /api/agent/token/commit
GET    /api/agent-upgrades/rollouts
POST   /api/agent-upgrades/rollouts/{id}/dispatch
POST   /api/agent-upgrades/rollouts/{id}/approve
POST   /api/agent-upgrades/rollouts/{id}/cancel
GET    /api/agent-upgrades/rollouts/{id}/outcomes

Orchestration

GET    /api/orchestration/meta
GET    /api/orchestration/approvals
GET    /api/workflows                  POST /api/workflows
GET    /api/runs                       POST /api/runs
GET    /api/runs/{id}
POST   /api/runs/{id}/approval

Discovery, monitoring and alerts

GET    /api/findings                   PATCH /api/findings/{id}/state
POST   /api/findings/{id}/import
GET    /api/monitor/tls                POST /api/monitor/tls
GET    /api/monitor/tls/{id}/history   POST /api/monitor/scan
GET    /api/monitor/tls/alerts         POST /api/monitor/tls/alerts/{id}/acknowledge
GET    /api/alerts
POST   /api/alerts/{id}/ack            POST /api/alerts/{id}/resolve

Identity and access

GET    /api/users                      POST /api/users
PATCH  /api/users/{id}                 DELETE /api/users/{id}
GET    /api/access/summary             GET  /api/access/catalog
GET    /api/access/directory           GET  /api/access/preview
POST   /api/access/teams               POST /api/access/owners
POST   /api/access/roles               POST /api/access/group-roles
GET    /api/access/ca-policy           POST /api/access/ca-policy
GET    /api/access/audit
GET    /api/service-accounts           POST /api/service-accounts
PATCH  /api/service-accounts/{id}
POST   /api/service-accounts/{id}/tokens
DELETE /api/service-accounts/{id}/tokens/{token_id}

Operations and settings

GET    /api/meta                       GET  /api/dashboard
GET    /api/settings                   PATCH /api/settings
GET    /api/selftest
GET    /api/events
GET    /api/notifications              GET  /api/notifications/settings
PATCH  /api/notifications/settings     POST /api/notifications/test
GET    /api/notifications/channels     POST /api/notifications/channels
POST   /api/notifications/channels/{id}/test
POST   /api/notifications/{id}/retry
GET    /api/renewals                   POST /api/renewals/run
POST   /api/renewals/{certId}/retry
GET    /api/orders                     POST /api/orders/{id}/continue
GET    /api/enterprise/deployment/backups
POST   /api/enterprise/deployment/backups
GET    /api/enterprise/deployment/recovery
Note: Routes under administrator-only prefixes (9.7) require an administrator session even when the caller holds a matching scope. Migration endpoints require the migration admin bearer and are disabled without it.

13.3Errors and status codes

Errors are JSON. Most carry a message; authorization failures also carry a code.

{ "code": "RBAC_DENIED", "message": "The role or owner team does not grant access to this operation" }
{ "message": "Unknown scope: certificates.superuser" }
StatusMeaningTypical cause
400Invalid requestBad field, unknown scope, invalid expiry, malformed CSR.
401Not authenticatedMissing or expired session, revoked or expired token, wrong secret.
403Not authorizedRole, grant, scope or administrator boundary; missing/incorrect CSRF; origin mismatch.
404Not foundUnknown id — also returned for resources outside your scope.
409ConflictDuplicate name, alert/rollout already actioned, EJBCA approval still pending, disabled account.
502Upstream failureThe CA or target rejected the request; the message preserves the CA's answer.
503Not readyDatabase, identity store or required configuration missing — check /readyz.
Tip: A 502 usually means the CA refused for a policy or permission reason. Read the message and fix the CA side; retrying the same request unchanged will fail the same way.

14.1What is verified

Everything below was executed against this source tree or a recorded lab environment. Anything not listed here is covered in 14.2 — treat it as unproven.

AreaEvidence
Rust API and consolecargo test: 283 passed, 0 failed, including authorization, session, connector, deployment, orchestration and service-account suites.
Go agentgo vet ./..., go test ./... and go test -race ./... pass.
Route classificationscripts/audit-rust-authorization.py: 219 route-method pairs, 0 needing explicit classification.
API parityscripts/audit-api-parity.py: 112 frontend paths, 112 registered routes, 0 missing (static check, not behavioural parity).
Test profile stackBoth images built, API healthcheck passed, console served; sign-in, settings, inventory, orders, events, issuance with retained key, idempotent continuation, 7/7 self-test and revocation executed through nginx.
Clientless SSHDelivery, verification and rollback executed end to end on the lab.
Clientless WinRMInstall, same-key upgrade and rollback verified in a Windows Server 2025 container; reload and TLS presentation scripts verified with five checks and zero failures.
AD CS issuanceWinRM issuance verified end to end against a Windows Server 2025 CA; Web Enrollment issuance verified against a domain CA with the role installed.
Windows PFXImport into LocalMachine/My with hasPrivateKey=true and a read ACE for a non-admin service identity, in a Server Core container; then use of that key by the bound identity, and refusal for an identity without the ACE, on a live Windows Server 2025 host.
IIS changeoutRebind a wildcard and an SNI HTTPS binding in a Windows Server 2025 IIS container, read HTTP.sys back, and fail an ambiguous binding without disturbing the real one. A site with no HTTPS binding was given one and served the import on a live handshake; a stopped site's new binding and HTTP.sys entry were removed when the job could not confirm the changeout. A leaf issued through an intermediate CA under the lab root was imported into an application site with its own app pool, served 200 with the application body, built a three-level chain, and survived an app-pool recycle. A queued changeout then recycled that pool itself — the worker pid changed, the old process drained and exited, and the site kept serving the same certificate.
WinRM PFX importRequest validation and script substitution are tested; a live import needs a reachable WinRM host with staged PFX material.
EJBCA connectorLab lifecycle: create, test, issue, renew and revoke through the REST API.
Vault PKI connectorVault 1.18.5: create, update, test, ECDSA P-256 issue, OpenSSL chain verification, renew with a fresh key, revoke and CRL check.
ACME v2Local stub: HTTP-01 issue and revoke, wildcard DNS-01 with TXT creation and cleanup, rejected TXT, rejected cleanup, and journal recovery after an API kill.
Revocation workerChecked a real Google Trust Services CRL and a live OpenSSL OCSP responder reporting both good and revoked.
ReplicasTwo API replicas split background work into distinct leases.
Backup and restoreDestructive drill restored certificates, accounts, audit events and connector credentials.
Vault rotationLive rotation rewrapped six sealed values with zero skips; connectors still opened them after the previous key was removed.
Service accountsScoped token reaches a granted read, is refused a classified write and four administrator surfaces, digest-only storage, closure on revoke/expiry/disable.
Verify: Re-run the local evidence yourself with bash scripts/release-gate.sh (14.3).

14.2What still needs your environment

Open itemWhy it is openWhat closes it
Live EST and SCEPNo production-like EST or SCEP authority was available.Issue, renew and revoke against your own EST/SCEP endpoints.
Public ACMEOnly local stubs and preflight checks were run.A real Let's Encrypt or commercial run with the port-80 route in place.
WinRM reload plus presentationOne Docker engine cannot run a Linux control plane beside Windows containers.One control-plane-driven deployment against a reachable Windows target.
PFX effective accessServer Core refuses non-admin logons (0xC0000142).Confirm the service identity can open the key on a real or domain-joined host.
WinRM PFX importNo reachable WinRM host with staged PFX material in the lab.Run the staged import and IIS changeout against a real WinRM target.
AD CS with your policyThe lab proves the transports, not your templates, issuance policy or RA rules.Issue against your template and confirm the returned chain.
Live OCSP with your issuerThe responder path is proven against a real OpenSSL responder in the suite, but the URL comes from the certificate, so only your CA's own responder proves your estate.Run the refresh against your issuer and confirm a revoked result reconciles that certificate.
Continuous presentation monitoring at scaleVerified per endpoint, not across a large estate.Run the monitor against your real endpoints for a full renewal cycle.
Independent security reviewNot performed.A signed review attestation accepted by the production gate.
Warning: The signed production gate reports these as blocking until a reviewer signs evidence. Do not treat a green staging gate as production approval.

14.3Release gate

# staging gate: tests, builds, source manifest, optional signature
bash scripts/release-gate.sh

# production gate: also requires a separately signed independent review
bash scripts/release-gate.sh --production

python3 scripts/verify-release.py
npm run typecheck
npm run verify:settings-route

What the stages check

StageChecks
Testscargo test --locked in the Rust backend, plus the repository's verification scripts.
Windows evidencescripts/windows-pfx-gate.sh runs the WinRM PFX end-to-end harness: a Windows Server container, a staged import over WS-Management, the HTTP.sys rebind and a live TLS handshake. A host that cannot run Windows containers records a skip with the reason; --production and the scheduled Windows job (SPIRYRA_REQUIRE_WINDOWS_E2E=1) refuse to pass on a skip, because the control plane runs on Linux but deploys certificates to Windows targets.
BuildsAPI binary and console build; no build-only secrets in the manifest.
ManifestSource digest recorded so a review can be bound to an exact tree.
Production reviewRefuses a review for a different source digest, an identical signing key, open high/critical findings or missing external-validation evidence.

verify:settings-route is a Playwright check for the canonical Identity Providers route and the legacy redirect. It needs a running test profile and either local administrator sign-in or the deployment migration admin token.

Verify: python3 scripts/verify-release.py reports the expected source digest and artifact list for the tree you intend to deploy.
Note: Key handling and the review attestation contract are specified in RELEASE_GATE.md. The signing key must be separate from the release signer.
Note: The Windows check needs a Windows host. The helper runs natively when a Windows container engine and python3 are on PATH (a Windows runner, or Docker Desktop in Windows-container mode), and otherwise re-enters WSL beside Docker Desktop's Windows engine. On a Linux runner it is recorded as skipped in reports/release-manifest.json and listed at the end of the gate output. Set SPIRYRA_REQUIRE_WINDOWS_E2E=1 to make a skip fail a staging run as well.
Note: CI keeps that skip honest. .github/workflows/ci.yml runs the gate on Linux for every push, and .github/workflows/windows-e2e.yml runs the same gate on a windows-2022 runner daily with SPIRYRA_REQUIRE_WINDOWS_E2E=1, so a skip or a regression fails a scheduled run instead of passing quietly.
Note: A schedule that stops firing cannot fail a job that never starts, so two watchdogs check the skipped evidence from outside, on every push and daily. .github/workflows/windows-evidence-watchdog.yml fails the run when the Windows workflow is missing, disabled, no longer declares a schedule: cron, or has no green run on the default branch within seven days. .github/workflows/console-evidence-watchdog.yml applies the same rule to the console workflow, except that a run is only owed after a push, so a quiet week is reported rather than failed. Both call scripts/check-evidence-freshness.py, read only the Actions API, and ping their own external heartbeat check so a watchdog that goes quiet still alerts.

15.1Installation and sign-in problems

SymptomCauseFix
production check fails on the originLoopback, IP address, localhost or a path in SPIRYRA_PUBLIC_ORIGIN.Use the exact browser-facing HTTPS DNS origin, no trailing path.
Preflight refuses the env filePermissions are not 0600, or tokens are equal or shorter than 32 characters.chmod 600 the file; regenerate distinct secrets.
/readyz returns 503Data volume, database or identity check failed.Read the failing check in the response and the startup log before restarting.
readyz says degradedNo CA connector configured yet.Expected on a fresh install; add and test a connector.
Sign-in says invalid credentials on a fresh test stackBootstrap hash not applied, or a reused database with a different password.Run reset-test-admin-password.sh (test only) or redeploy with the intended password.
Every write returns 403Missing CSRF header, or the proxy rewrote Host so the origin check fails.Send X-CSRF-Token; preserve Host in the proxy.
Console loads but API calls 502API container unhealthy or not started.docker compose -p <project> ps and inspect the api logs.
Browser warns about an insecure cookieTest profile over HTTP.Expected in test; never enable the lab switch in production.

15.2Connector problems

SymptomLikely causeFix
Test fails with a TLS errorPrivate root not in the connector trust bundle.Add the root PEM to deploy/ca/ and reference it; do not look for a skip-verify switch (there is none).
EJBCA test passes but issue failsMissing create-certificate or end-entity-profile rule.Grant the rules listed in 4.2.
Revoke returns 502, certificate still activeMissing revoke rule at the CA.Grant revoke on the CA and end-entity profile, then retry.
Issue rejected: “requested SAN is missing”The CA profile strips or forbids the SAN.Change the profile/role, or request only permitted names.
Vault issue rejected on TTLRole max_ttl shorter than requested.Request within the role's limit; the CA is authoritative.
ACME wildcard refusedConnector has no DNS zone, so only HTTP-01 is available.Configure dnsZoneId, dnsZoneName and dnsApiToken.
ACME HTTP-01 times outPort 80 not forwarded to the web container.Add the /.well-known/acme-challenge/ route in the proxy.
EST/SCEP unavailableHost not in the allowlist or the profile was not redeployed.Set SPIRYRA_EST_ALLOWED_HOSTS / SPIRYRA_SCEP_ALLOWED_HOSTS and redeploy.
AD CS Web Enrollment test failsAccount cannot enrol against the template, or the site certificate is untrusted.Grant enroll permission; set caPem to the site's trust anchor.
Tip: Always run Test first. The test isolates authentication, trust and CA visibility from the issuance request itself, which makes the fix obvious.

15.3Deployment problems

SymptomCauseFix
Delivery stays queuedAgent offline. Jobs do not replay automatically.Bring the agent online; the queued job is dispatched on its next poll.
copy_file deniedPath outside SPIRYRA_INSTALL_ROOTS.Widen the roots on the host, or choose a path inside them.
reload deniedService not in SPIRYRA_APPROVED_SERVICES.Add the service on the host; it is not a control-plane setting.
Probe deniedEndpoint not in SPIRYRA_TLS_PROBE_ALLOWLIST (empty denies all).Add the exact host:port on the host.
File digest mismatchWrite partially failed, or something rewrote the file.Retry; inspect the destination and any local automation on the host.
Fingerprint mismatch after reloadService did not actually reload, or a different certificate is served.Check the service and the endpoint's SNI; an automatic rollback is queued when the key did not change.
WinRM connection refusedWinRM disabled, or 5985 used in a profile that forbids HTTP.Enable WinRM; use 5986 with a trusted listener in production.
PFX import fails on ACLService identity unresolvable, or the ACE could not be written.Fix SPIRYRA_WINDOWS_PFX_SERVICE_ACCOUNT; the job fails closed by design.
Rollback marks restored but the service serves the old certReload was not approved or did not run.Re-issue the reload, then verify the presented fingerprint.

15.4Orchestration problems

SymptomCauseFix
A WinRM server is reported as not runnableThe host is not in SPIRYRA_AGENTLESS_WINRM_HOSTS, or the password, username or host is missing.Add the host to the API-host list and store the sealed password on the server.
WinRM step is refused with an allowlist messageThe path, command, service or endpoint is not in the SPIRYRA_WINRM_* list.Change the step or widen the API-host list — commands match exactly and paths must sit inside a listed root.
SSH step fails before it connectsSPIRYRA_AGENTLESS_SSH_KNOWN_HOSTS is unset, or the server's host key is not pinned.Pin the key on the API host; a non-default port needs the [host]:port form.
SSH step is refused with an allowlist messageThe path, command, service or endpoint is not in the SPIRYRA_SSH_* list.Change the step or widen the API-host list — the match is literal.
Discovery step is refused with 409Discovery needs the agent's inventory walk.Assign an agent to that server, or drop the discovery step.
Run waits indefinitelyAn approval gate is pending, or a host step's agent is offline.Check Approvals; gates time out rather than proceeding.
Command step fails before executionThe command does not exactly match SPIRYRA_APPROVED_COMMANDS.Change the step or the host allowlist — the match is literal.
Batch did nothingA target was invalid, so atomic creation refused the whole batch.Fix the target; the refusal is intentional.
Batch halted mid-wayA target failed and siblings were stopped before their next step.Read the batch detail: restored, reloaded and manual lists.
Recovery ledger still shows manual stepsCommands and API calls have no automatic inverse.Review each entry and close it by hand; it will not clear itself.
Run restarted from the beginningDatabase lease lost, or a second replica took over.Check logs for lease recovery; succeeded steps should be preserved.

15.5Diagnostic commands

# Stack state and logs
./scripts/deploy-profile.sh production status
docker compose -p spiryra-production ps
docker compose -p spiryra-production logs --tail=200 api

# Liveness and readiness
curl -fsS https://spiryra.example.com/healthz
curl -sS -i https://spiryra.example.com/readyz
curl -sS https://spiryra.example.com/api/meta | head -c 400

# Backups
./scripts/backup-profile.sh production
ls -1 backups/ | tail -3

# Recovery rehearsal (short outage)
./scripts/verify-production-deploy.sh --restore-drill --yes

# Authorization and parity audits
python3 scripts/audit-rust-authorization.py
python3 scripts/audit-api-parity.py
Note: /api/meta intentionally reports productionReady: false in this preview. A true value would mean the release gate passed, not a configuration checkbox.

16Glossary

ACME — Automatic Certificate Management Environment (RFC 8555). Spiryra supports HTTP-01 and DNS-01, including wildcards, with optional External Account Binding.

Agent — the native Go executable installed on a managed server. It polls the control plane, writes certificate files, reloads approved services and reports the certificate a socket presents.

Agentless (clientless) — delivery over SSH or WinRM with no Spiryra software on the target.

Allowlist (agent) — host-side lists that decide what a deployment may write, run, reload or probe. The control plane cannot widen them.

Approval gate — a workflow step that stops a run until an administrator approves or denies it.

Batch — one workflow run across several targets, created atomically.

CA policy — the record of which templates and key/validity constraints a deployment may use.

Connector — the configured path to an external authority (EJBCA, Vault PKI, ACME, AD CS, EST, SCEP), with sealed credentials.

CSRF token — a per-session value required on browser-originated writes (X-CSRF-Token).

Deployment target (installation) — a certificate bound to a server and destination path, with optional service reload and TLS endpoint.

EAB — External Account Binding, an ACME mechanism binding an account to a provider-issued key identifier and HMAC key.

EST — Enrolment over Secure Transport (RFC 7030).

Fingerprint — the SHA-256 digest of a certificate (or file), used to prove what was written and what is served.

Grant — a per-resource permission binding an account to a certificate or connector with read or manage rights.

Installation roots — the directories an agent may write into (SPIRYRA_INSTALL_ROOTS).

OCSP / CRL — the two revocation status mechanisms the worker polls for active certificates.

PFX / PKCS#12 — the Windows certificate container the agent can import into the machine store.

Readiness — /readyz: whether the deployment can actually do its job, not merely whether the process is alive.

Recovery ledger — the record of batch actions that have no automatic inverse, kept for operator follow-up.

Retained key — a private key stored sealed in the vault, enabling key-reuse renewal and administrator-gated download. Never deployed.

Run — one workflow execution against one target, with per-step state and evidence.

SCEP — Simple Certificate Enrolment Protocol (RFC 8894). One common name per request, no revocation.

Scope (account) — all or assigned; combined with grants to decide which resources an account may act on.

Scope (service account) — an action from the permission catalogue granted to a machine token; matched exactly.

Sealed value — a secret encrypted with AES-256-GCM under SPIRYRA_VAULT_KEY.

Vault key — the wrapping key for every sealed value. Back it up separately from data backups.

Workflow — the ordered definition of steps, targets and conditions that a run executes.

17Document index

Every claim in this guide is backed by a repository document. Start with the first two rows for status and deployment.

DocumentContents
NATIVE_RELEASE_STATUS.mdVerification passes, gaps and release classification.
DEPLOY_FRESH.mdProfiles, manual env, TLS proxy, backup, rotation, removal.
RELEASE_GATE.mdSigned staging and production gate, manifest and review contract.
SERVICE_ACCOUNTS.mdMachine credentials, scopes, admin API and tests.
EXTERNAL_CA_ONLY.mdWhy Spiryra is an interface to your CAs, not a CA.
RUST_EJBCA_CONNECTOR_MIGRATION.mdEJBCA REST lifecycle, roles and validation.
RUST_VAULT_PKI_CONNECTOR.mdVault PKI connector and lab proof.
RUST_ACME_CONNECTOR.mdACME v2, HTTP-01, EAB, DNS-01 and rechecks.
CLIENTLESS_ADCS_TRANSPORTS.mdAD CS over WinRM and Web Enrollment, with evidence.
ACME_DNS01_EST_WORK_STATUS.mdEST work status and DNS-01 detail.
SCEP_AGENTLESS_IMPLEMENTATION_STATUS.mdSCEP implementation and limits.
AGENTLESS_DEPLOYMENT_E2E.mdSSH and WinRM delivery, verification and rollback evidence.
AGENTLESS_SSH_WINRM_STATUS.mdCurrent agentless status and remaining checks.
WINDOWS_CONTAINER_TEST_EVIDENCE.mdWindows Server 2025 container evidence for WinRM and PFX.
WINDOWS_PFX_IMPLEMENTATION.mdPFX import and service-account ACL binding.
RUST_AGENT_ENROLLMENT_MODULE.mdAgent registry, enrollment, rotation and limits.
AGENT_UPGRADE_DISPATCH_STATUS.mdSigned release catalog and supervisor.
RUST_ORCHESTRATION_MODULE.mdCanvas, runs, parallel groups, approvals and batches.
RUST_RENEWAL_DEPLOYMENT.mdRenewal plus deployment evidence and timeouts.
RBAC_SECURITY_WORK_COMPLETED.mdRoles, teams, ownership, scoped reads, admin boundary.
RBAC_ADMIN_BOUNDARY_REVIEW.mdWhy each administrator surface and exception exists.
PASSWORD_MANAGEMENT_STATUS.mdPassword policy, lockout, history and recovery.
SMTP_ESCALATION_E2E_PROOF.mdNotification delivery and escalation evidence.
CONNECTOR_E2E_PROOF.mdConnector end-to-end lab proof.
RELEASE_VERIFICATION_2026-10-30.mdWhat was executed for the current closeout, and what was not.
IMPLEMENTATION_PROGRESS.mdProgress narrative across the native migration.
Note: This HTML guide and the Markdown documents are generated from the same source tree. When they disagree, the Markdown evidence wins — then update this guide.