1.1About this documentation
This guide describes how to install Spiryra, connect the certificate authority you already operate, and use every feature in the console: certificate lifecycle, agents and clientless deployment, orchestration, identity, monitoring, service accounts and operations. Spiryra is published by Nacreon Security at nacreonsecurity.com.
Nacreon Security ships the Nacreon suite, of which Spiryra is the certificate lifecycle product. The suite, its documentation and the support policy are published at nacreonsecurity.com; this guide is the manual for one product in it.
Who this is for
Platform and security engineers who run the Docker host, PKI administrators who own the external CA, and operators who issue, deploy and renew certificates day to day. Installation chapters assume shell access to the Docker host; feature chapters assume a signed-in administrator.
How the chapters are arranged
| Chapters | What you do there |
|---|---|
| 1–3 | Understand the runtime, install a profile, sign in and complete the first-run checklist. |
| 4–5 | Connect an external CA, then import, issue, renew and revoke certificates. |
| 6–7 | Deliver certificates to Linux and Windows targets, then automate the work with workflows. |
| 8–10 | Discover and monitor, control who can do what, and route notifications. |
| 11–13 | Operate the deployment, review the security model and call the API. |
| 14–17 | Read the verification status, troubleshoot, and look up terms and source documents. |
Conventions
- Monospaced text is a console label, file path, environment variable or literal value.
- Code blocks run on the Docker host from the repository root unless the heading says otherwise.
- /api/… paths are API routes; most have an equivalent screen in the console.
- Commands that change data or delete volumes are marked destructive.
Callout styles
1.2Edition comparison and release status
Both editions use the same images and application code. They differ only in runtime configuration, data volume, ports and safeguards.
| Test profile | Production profile | |
|---|---|---|
| Purpose | Evaluation, UI work, connector and agent experiments, orchestration and RBAC testing. | A hardened runtime against the organization's existing CA. |
| Network | Loopback HTTP, or one explicit LAN address on a trusted lab network. | Loopback HTTP behind your HTTPS reverse proxy; exact public origin. |
| Cookies | SPIRYRA_LAB_INSECURE_HTTP_COOKIE=true | Secure cookies; the lab switch is refused. |
| Certificate authority | Optional disposable EJBCA Community CA for evaluation. | External CA only. No local CA, no demo seed. |
| Administrator | admin / Testing123, reapplied on every deployment. | Generated bootstrap credential, printed once; change after sign-in. |
| Data volume | spiryra-test_spiryra-data | spiryra-production_spiryra-data |
| Compose project | spiryra-test | spiryra-production |
What “development preview” means
The Rust API and console build, the test stack runs, and the verification described in 14.1 was executed. It does not mean the product is certified for production certificate management. The release gate deliberately blocks until live EST/SCEP and public ACME runs and an independent security review are complete; the WinRM orchestration steps have since run on a real Windows endpoint (WINDOWS_LIVE_WINRM_E2E.md).
Source-of-truth documents
| Document | Contains |
|---|---|
| NATIVE_RELEASE_STATUS.md | Verification evidence, known gaps, release classification. |
| DEPLOY_FRESH.md | Profile commands, manual env workflow, backup, rotation. |
| RELEASE_GATE.md | Signed staging and production gate contract. |
| SERVICE_ACCOUNTS.md | Machine credentials, scopes and the service-account API. |
1.3Deployment architecture
The console and API run in separate containers. The browser never reaches the API container directly: nginx serves the single-page console and proxies /api/ to the API over the internal network.
Runtime boundaries
- Web container: nginx on port 8080, bound to 127.0.0.1 by default. In production your proxy publishes HTTPS.
- API container: internal port 9080, health-checked, never published.
- Data: one Compose volume per profile holding SQLite databases, the optional keyring, the ACME webroot and deployment evidence.
- Secrets: deploy/<profile>.env, mode 0600, never committed.
- Vault: AES-256-GCM wrapping for connector credentials, retained leaf keys, SSH keys, WinRM passwords and notification secrets.
What the API talks to
| Target | Direction | Requirement |
|---|---|---|
| External CA | Outbound only | HTTPS with a trust bundle you configure, or WinRM/Web Enrollment for AD CS. |
| Agents | Inbound polling | Agent polls /api/agent/jobs; no inbound port is opened on the target. |
| SSH / WinRM targets | Outbound | TCP 22, or WinRM 5985 (lab) / 5986 (HTTPS). |
| SMTP / webhooks | Outbound | HTTPS webhooks only; private and loopback targets blocked unless explicitly enabled. |
| Public ACME | Inbound port 80 | /.well-known/acme-challenge/ forwarded to the web container. |
1.4Supported platforms and prerequisites
Docker host
| Requirement | Why | Check |
|---|---|---|
| Docker Engine with Compose v2 | Runs the API, web console and optional test CA. | docker compose version |
| Python 3 with argon2-cffi | Generates the local administrator Argon2id hash. | python3 -m pip install --user argon2-cffi |
| openssl | Random secrets and test CA material. | openssl version |
| curl | Health checks used by the deploy and verify scripts. | curl --version |
| Node.js 18+ and Playwright (optional) | Automatic Settings-route browser check in the test profile. | npm install, npx playwright install --with-deps chromium |
| TLS reverse proxy (production) | HTTPS termination for the public origin. | nginx, Caddy, HAProxy or your standard ingress. |
Managed targets
| Path | Supported platforms | Requirements on the target |
|---|---|---|
| Native agent | Linux x86-64, Linux ARM64, Windows x86-64 | Outbound HTTPS to the console, a writable token path, and a service reload allowlist if you reload services. |
| Clientless SSH | Linux and Unix-like hosts | SSH user with write access to the certificate path and, optionally, a permitted reload command. |
| Clientless WinRM | Windows Server | WinRM enabled with Auth\Basic on (Windows disables it by default); HTTPS (5986) with a trusted listener in production. |
Certificate authorities
| Connector | Minimum on the CA side |
|---|---|
| EJBCA REST | EJBCA 7+ with the REST API enabled and an identity holding the CA, RA and end-entity-profile rules listed in 4.2. |
| HashiCorp Vault PKI | Vault with a PKI secrets engine and a signing role (lab-verified on 1.18.5). |
| ACME v2 | An RFC 8555 directory URL; EAB if your provider requires it; DNS API access for wildcards. |
| Microsoft AD CS | A CA with a certificate template, reachable over WinRM or with the Web Enrollment role installed. |
| EST | An RFC 7030 EST endpoint; the host must be listed in SPIRYRA_EST_ALLOWED_HOSTS. |
| SCEP | An RFC 8894 SCEP endpoint; the host must be listed in SPIRYRA_SCEP_ALLOWED_HOSTS. |
Browsers
The console is a single-page application tested in current Chromium-based browsers and Firefox. It requires JavaScript and secure cookies in the production profile.
2.1Install the test profile
The test profile is the recommended first install. It deploys Spiryra with an optional disposable EJBCA Community CA, so you can exercise the whole lifecycle without touching a real authority.
Run the wizard
./scripts/deploy-test.sh
The wizard asks for the bind address, port and administrator username, then writes deploy/test.env with mode 0600, starts the stack, waits for health and prints the console URL and administrator credential.
What it creates
- Spiryra on HTTP, loopback by default, or one explicit --bind address.
- A disposable EJBCA Community CA from compose.test-ca.yaml, when enabled.
- EJBCA client certificate, private key and ManagementCA bundle in deploy/test-ejbca/ (secrets mode 0600).
- A paste-ready deploy/test-ejbca/connector-paste-values.txt for the Connectors screen.
- A reusable deploy/test.env carrying SPIRYRA_TEST_CA=true, so later profile commands keep the CA container in the same lifecycle.
Flags
| Flag | Use it when |
|---|---|
--check | Validate prerequisites and the existing env without touching containers. |
--yes | Unattended deployment; generated credentials are printed once. |
--register-connector | Create and test the EJBCA Test CA connector through the API instead of pasting values. |
--no-ejbca | Spiryra only, no disposable CA. |
--bind 192.168.40.204 | Lab access from another machine. HTTP only — never use this beyond a trusted network. |
--skip-ui-check | The host has no Node or Playwright. |
--fresh | Destructive: replace test containers and the data volume. |
--admin-password VALUE | Override the test administrator; also SPIRYRA_TEST_ADMIN_PASSWORD. |
The test credential
Sign in as admin with Testing123 unless you overrode it. The password is reapplied after every test deployment, and deploy-production.sh refuses it. To reset a running stack without redeploying:
SPIRYRA_TEST_ADMIN_PASSWORD=Testing123 ./scripts/reset-test-admin-password.sh
curl -fsS http://127.0.0.1:8080/healthz returns ok, and curl -i http://127.0.0.1:8080/readyz returns 200. A fresh install may report degraded until a CA connector is configured — that is expected, not a failure.2.2Install the production profile
The production profile deploys Spiryra only. It never starts a CA and never seeds demo data; you connect the organization's existing CA after sign-in.
./scripts/deploy-production.sh \
--origin https://spiryra.example.com
# preflight only, no containers touched
./scripts/deploy-production.sh --check
The origin must be an exact public HTTPS DNS origin: no path, query, IP address or localhost. The wizard generates a unique bootstrap administrator credential and prints it once.
What the preflight refuses
- SPIRYRA_DEPLOYMENT_PROFILE not set to production.
- A loopback, IP-address or non-HTTPS SPIRYRA_PUBLIC_ORIGIN.
- SPIRYRA_LAB_INSECURE_HTTP_COOKIE=true, or any SPIRYRA_LAB_ALLOW_* switch.
- The TLS-verification opt-outs SPIRYRA_ADCS_WEB_INSECURE=true or SPIRYRA_AGENTLESS_WINRM_INSECURE=true (also refused as 1).
- SPIRYRA_METRICS_TOKEN missing, shorter than 32 characters, or still a template placeholder.
- An env file readable or writable by group or world.
- Migration tokens or a vault key shorter than 32 characters, or values that are not distinct.
- The lab default password Testing123.
- An API that advertises a local-ca kind or demoAvailable: true, or a Compose file such as compose.test-ca.yaml.
After the stack is healthy
- Put TLS in front of the console. Proxy your public origin to http://127.0.0.1:8080; see 2.4.
- Sign in with the bootstrap credential and change it immediately.
- Connect the existing CA under Connectors and test it before issuing anything.
- Review policy under Settings: expiry windows, default validity, key rules, SAN policy, session length.
- Configure notifications and run a backup; confirm /readyz is ready.
./scripts/deploy-profile.sh production status reports healthy containers, and curl -i https://spiryra.example.com/readyz returns 200 with the CA connector check ok.2.3Manual environment installation
Use this when you own the env file instead of letting the wizard generate it — for example when secrets come from your secret manager.
- Create the env file and lock it down.
cp deploy/test.env.example deploy/test.env chmod 600 deploy/test.env - Generate secrets. Three distinct migration tokens and one vault key:
openssl rand -hex 32 # run three times for the migration tokens openssl rand -hex 32 # once for SPIRYRA_VAULT_KEY - Generate the administrator hash.
Paste the complete single-quoted SPIRYRA_LOCAL_ADMIN_ARGON2_HASH=… line it prints.python3 scripts/create-lab-admin.py - Replace every remaining placeholder (REPLACE_WITH…) in the env file.
- Validate and start.
./scripts/deploy-profile.sh test check ./scripts/deploy-profile.sh test up ./scripts/deploy-profile.sh test status
Production follows the same pattern with deploy/production.env and the production profile. production check fails closed until the origin, cookies, tokens and vault key are valid.
Profile commands
./scripts/deploy-profile.sh <test|production> check
./scripts/deploy-profile.sh <test|production> up
./scripts/deploy-profile.sh <test|production> status
./scripts/deploy-profile.sh <test|production> digest
./scripts/deploy-profile.sh <test|production> down
start is accepted as an alias for up; both go through the same checks.
Stale images are refused, not served
A profile runs whatever images it was started from, and a container keeps its name and its Up (healthy) status while the sources move on. Each image is therefore stamped at build time with a SHA-256 over the files it is built from — the Dockerfile, the manifests, the sources, and for the API the agent binaries it serves — and the stamp is compared with the tree whenever someone asks.
status and doctor print the verdict after the container table and exit non-zero on drift; digest reports it on demand. up refuses to start images built from a different tree, because starting them is how the drift becomes serving, and up --allow-stale overrides that deliberately. An image built before the stamp existed cannot be compared at all, so it warns and starts instead of blocking an installation that predates the check.
./scripts/deploy-profile.sh production status # non-zero when the images drifted
./scripts/deploy-profile.sh production build
./scripts/deploy-profile.sh production up # refuses a drifted image
Generated material
| Path | Contents | Handling |
|---|---|---|
| deploy/<profile>.env | Migration tokens, vault key, bootstrap hash. | 0600. Back up with your secret manager, separately from data backups. |
| deploy/test-ejbca/ | Client certificate, key, ManagementCA bundle, PKCS#12. | Disposable lab material only. |
| deploy/ca/ | Trust bundles mounted read-only into the API at /etc/spiryra/ca. | PEM only; referenced by env, never uploaded through the UI. |
| backups/<profile>-<timestamp>/ | Verified SQLite snapshots plus a checksummed manifest. | Contains sealed secrets but not the vault key. |
2.4Reverse proxy and TLS
The web container serves HTTP on loopback. Your proxy terminates TLS for the public origin and forwards to it, preserving the browser's Host header.
Checklist
- Terminate HTTPS for the exact SPIRYRA_PUBLIC_ORIGIN.
- Proxy to http://127.0.0.1:8080; never publish that port.
- Preserve Host (including a non-default port) and send X-Forwarded-Proto: https.
- Forward /.well-known/acme-challenge/ on port 80 to the web container for public ACME HTTP-01.
- Allow request bodies large enough for certificate imports and workflow payloads (a few MB is sufficient).
nginx example
server {
listen 443 ssl;
server_name spiryra.example.com;
ssl_certificate /etc/ssl/certs/spiryra.pem;
ssl_certificate_key /etc/ssl/private/spiryra.key;
location / {
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
}
}
# ACME HTTP-01 only
server {
listen 80;
server_name spiryra.example.com;
location /.well-known/acme-challenge/ { proxy_pass http://127.0.0.1:8080; }
location / { return 301 https://$host$request_uri; }
}
TLS to external systems
Connector and webhook TLS is verified with the trust bundle you configure. Redirects are refused, embedded credentials in URLs are rejected, and no “skip verification” switch exists. Place private roots in deploy/ca/, mounted read-only at /etc/spiryra/ca, and reference the file by absolute path.
2.5Environment variable reference
All variables live in deploy/<profile>.env and are passed to the API container by Compose. Values marked secret are never printed by the console.
Core
| Variable | Purpose |
|---|---|
| SPIRYRA_DEPLOYMENT_PROFILE | test or production. Production refuses any other value. |
| SPIRYRA_PUBLIC_ORIGIN | Exact browser-facing HTTPS origin. Used for links, cookies and same-origin checks. |
| SPIRYRA_DOCS_URL | Documentation link in the console sidebar footer. Defaults to /DEPLOYMENT_GUIDE.html, which the web image publishes beside the console. Set an https:// URL to point at your own copy. |
| SPIRYRA_METRICS_TOKEN | Secret. When set, /metrics requires Authorization: Bearer with this value. Required in production; with it empty the scrape endpoint answers unauthenticated to anything on the container network, so the preflight refuses a production env without one. |
| SPIRYRA_LOGIN_RATE_LIMIT / …_WINDOW_SECS | Sign-in attempts allowed per client address in one window. Default 10 per 300 seconds; 0 disables the bucket. |
| SPIRYRA_ENROLL_RATE_LIMIT / …_WINDOW_SECS | Self-enrollment attempts allowed per client address. Default 30 per 3600 seconds. Both windows are stored in the API database, so replicas that share that file spend one budget between them. |
| SPIRYRA_TRUSTED_PROXIES | Which peer may speak for the client. Unset or none uses the socket address and ignores X-Real-IP/X-Forwarded-For; * trusts the header from whatever connects; a comma-separated address or CIDR list (for example 10.0.0.0/8,192.168.40.24) trusts it only from those peers. The bundle defaults to the Docker bridge range, because the bundled web container is the only path to the API. |
| SPIRYRA_WINDOWS_PFX_KEEP_STAGED / SPIRYRA_WINRM_PFX_KEEP_STAGED | true keeps the staged PFX and password ciphertext after a verified import. Default: they are deleted, so private key material does not linger. A failed import always leaves them in place. |
| SPIRYRA_WEB_BIND | One explicit host address for the web container. Loopback by default; LAN binding is test-only. |
| SPIRYRA_LAB_INSECURE_HTTP_COOKIE | Test-only switch that allows cookies over HTTP. Refused in production. |
| SPIRYRA_VAULT_KEY | Secret. AES-256-GCM wrapping key for every sealed value. Losing it makes sealed data unrecoverable. |
| SPIRYRA_VAULT_KEY_PREVIOUS | Secret. Comma-separated previous keys accepted during rotation. |
| SPIRYRA_MIGRATION_ADMIN_TOKEN | Secret. Bootstrap administrator bearer used by scripts and recovery. |
| SPIRYRA_MIGRATION_READER_TOKEN | Secret. Read-only migration credential. |
| SPIRYRA_MIGRATION_REVIEWER_TOKEN | Secret. Review-only migration credential. |
| SPIRYRA_LOCAL_ADMIN_ARGON2_HASH | Secret. Argon2id hash of the bootstrap administrator password. |
Connectors
| Variable | Purpose |
|---|---|
| SPIRYRA_ACME_WEBROOT | Directory for HTTP-01 challenge files, shared with nginx. |
| SPIRYRA_EJBCA_ISSUED_CA_BUNDLE | PEM bundle used to verify certificates returned by EJBCA. Empty disables issuance verification. |
| SPIRYRA_EJBCA_APPROVAL_POLL_SECS | Seconds between polls of an EJBCA enrollment waiting for RA approval. Default 300, allowed 5–86400. |
| SPIRYRA_EJBCA_APPROVAL_MAX_POLLS | Failed polls before that request is marked expired. Default 72, about six hours at the default interval. |
| SPIRYRA_EST_ALLOWED_HOSTS | Comma-separated EST hostnames. Empty keeps EST unavailable. |
| SPIRYRA_SCEP_ALLOWED_HOSTS | Comma-separated SCEP hostnames. Empty keeps SCEP unavailable. |
| SPIRYRA_AGENTLESS_WINRM_ALLOW_HTTP | Lab only. Permits WinRM over 5985. Production should use HTTPS 5986. |
Agent plane
| Variable | Purpose |
|---|---|
| SPIRYRA_RUST_FLEET_SQLITE | Absolute path to the fleet database. Without it the agent plane answers 503 — there is no in-memory fallback. |
| SPIRYRA_AGENT_ENROLLMENT_TOKEN | Secret. ≥32-character campaign secret enabling self-enrollment. |
| SPIRYRA_AGENT_CAMPAIGN_TENANT | Tenant a campaign token enrolls into (lowercase identifier). Empty means the default tenant; an invalid value refuses enrollment rather than enrolling a host into the wrong inventory. |
| SPIRYRA_AGENT_CURRENT_VERSION | Expected agent build label, default go-preview. |
| SPIRYRA_AGENT_DOWNLOADS_DIR | Directory holding agent binaries served by the downloads route. |
| SPIRYRA_AGENT_RELEASE_CATALOG | Local JSON catalog of signed releases for staged upgrades. |
| SPIRYRA_APPROVED_SERVICES | Agent-side allowlist of services a deployment may reload. |
| SPIRYRA_TLS_PROBE_ALLOWLIST | Agent-side allowlist of TLS endpoints that may be probed. Empty denies all. |
| SPIRYRA_TLS_PROBE_MAX_TIMEOUT_SECS | Local cap on the requested probe timeout. |
| SPIRYRA_WINDOWS_PFX_SERVICE_ACCOUNT | Service identity granted read access to the imported PFX private key (SAM or UPN). |
| SPIRYRA_WINDOWS_PFX_REQUIRE_SERVICE_ACCOUNT | When true, refuse a PFX import that has no configured identity. |
| SPIRYRA_UPDATE_SIGNING_PUBLIC_KEY | Pinned public key agents use to verify staged upgrade signatures. |
2.6Upgrade an installation
- Read the change notes for the release, then back up the profile:
./scripts/backup-profile.sh production - Check out the new source on the Docker host (same repository, same profile env file).
- Validate the configuration before replacing containers:
./scripts/deploy-profile.sh production check - Deploy. The stack recreates the API and web containers; schema migrations run on startup.
./scripts/deploy-profile.sh production up - Confirm readiness, then sign in and spot-check inventory, a connector test and one deployment target.
/healthz answers, /readyz returns 200 with the database and identity checks ok, and the console shows the expected certificate count.Agent upgrades are separate
Agent binaries are upgraded through the signed release catalog, not by redeploying the control plane. See 6.4. An agent that is offline keeps its job queued; deployments do not replay automatically.
Rolling back an application upgrade
Redeploy the previous source revision. If a migration changed the schema, restore the pre-upgrade backup instead of downgrading binaries:
./scripts/restore-profile.sh production backups/production-20261031T020304Z --yes
2.7Remove a profile
./scripts/deploy-profile.sh test down
./scripts/deploy-profile.sh test down --volumes --yes # destructive
down stops containers and keeps the data volume. Adding --volumes --yes removes the profile's containers and volume — all certificates, accounts, audit history and sealed credentials for that profile.
For the test profile, ./scripts/deploy-test.sh --fresh also removes the disposable EJBCA containers and the ejbca-data volume. Production never has a test CA to remove, and profile commands never touch the other profile's data.
3.1Bootstrap credentials and sign-in
| Profile | Credential | Behaviour |
|---|---|---|
| Test | admin / Testing123 | Prompted on every interactive deployment; reapplied after the stack is healthy. Lab only. |
| Production | Generated bootstrap password | Printed once at deployment. Change it after the first sign-in. |
Sign-in flow
- POST /api/auth/login returns a session cookie plus a CSRF token bound to that session.
- Every browser-origin write must send the CSRF token; a missing or wrong value returns 403.
- Bearer credentials — migration tokens, agents and API service accounts — are not subject to the browser CSRF rule.
- Session lifetime follows the configured policy (sessionHours in Settings).
First actions
- Change the bootstrap password under your account menu.
- Create one administrator account per person; stop sharing the bootstrap credential.
- Configure the CA connector, then policy, then notifications.
- Run a backup and store the vault key in your secret manager.
3.2First-run checklist
Work through this list in order. The first issued and deployed certificate is the fastest proof that the connector, policy, target and verification path all agree.
- Sign in and change the bootstrap credential (3.1).
- Connect the external CA under Connectors and press Test. Do not issue until the test passes (4.1).
- Set policy under Settings: expiry warning and critical windows, default validity, key rules, SAN policy, session length (5.7).
- Issue or import the first certificate and confirm it appears in inventory with the expected issuer and SANs (5.2).
- Register a deployment target — enrolled agent, SSH host or WinRM host — and confirm it is ready (6.1).
- Deploy the certificate to an absolute path with an optional approved reload and TLS endpoint (6.3).
- Verify presentation: the deployment only becomes verified when the fingerprint the socket serves matches the file (6.3).
- Close the loop: notifications, escalation, a profile backup and a green /readyz (10 Notifications, 11.2).
3.3Password policy and account recovery
Policy
- Local passwords are stored as Argon2id hashes; sessions store only a SHA-256 digest of the session token.
- Policy covers minimum length and complexity, history, maximum age and lockout after repeated failures.
- Policy is stored in the database and read by the sign-in path — a failed account-schema migration reports 503 instead of silently breaking sign-in.
- Changing a password revokes existing sessions; an expired password is changed through the dedicated expired-password route.
Self-service recovery
Recovery requires an SMTP relay and a recovery email on the account. The administrator issues a one-time token, the user receives a link, and the token is single-use with a short expiry.
POST /api/auth/password/recover
POST /api/auth/password/change
POST /api/auth/password/change-expired
GET /api/auth/password/policy
Administrators with the users.manage action can also reset a password directly, which is the fallback when mail is unavailable.
4.1Connector model and the credential vault
Spiryra is an interface to the certificate authorities you already operate. It never hosts an issuing CA in production. A connector is the configured, tested, audited path to one authority.
How a connector works
- Non-secret settings (URLs, CA names, profiles, usernames) are stored as connector configuration and returned by the API.
- Secrets — private keys, tokens, passwords, trust bundles — are sealed into one AES-256-GCM document bound to the connector id. The API returns only the names of stored secret fields, never the values.
- Tests are explicit. A connector is not “working” until Test succeeds; the console shows the outcome and the audit log records it.
- Issuance stays authoritative at the CA. Spiryra generates the key and CSR, then validates what comes back before anything enters inventory.
Add and test a connector
- Open Connectors → Add connector and choose the type. The form fields come from the connector specification (GET /api/connectors/specs).
- Fill the non-secret settings and paste or upload the secrets.
- Save, then press Test. The test authenticates, reads the CA, and reports what it found — for example the issuing CA and the available templates.
- Optionally run Pull profiles from EJBCA (or the equivalent discovery for that connector) to import the authority's template list for the issue form and CA policy.
Returned-certificate validation
Before a certificate is stored, Spiryra checks: the subject CN matches the request, every requested SAN is present, the public key matches the key that generated the CSR, the certificate is not expired, and the signature verifies against your configured trust bundle. The chain is ordered with the issuing CA first.
Housekeeping
- Rotate connector credentials after vault rotation (11.3) and after any suspicion of exposure.
- Disable rather than delete a connector that has issuance history; disabled connectors stay in audit records.
- Every CA-side rejection is surfaced as an HTTP error with the CA message; the inventory is only updated after the CA confirms.
4.2EJBCA REST connector
EJBCA is reached over its REST API with mutual TLS, a bearer token or basic authentication. Spiryra generates the key and PKCS#10 request locally and calls /certificate/pkcs10enroll.
Configuration
| Field | Type | Notes |
|---|---|---|
| baseUrl | URL | EJBCA HTTPS origin. Non-HTTPS origins, embedded credentials, paths, queries and fragments are rejected; redirects are disabled. |
| caName | text | Issuing CA. The test proves this CA exists. |
| certificateProfile | text | Certificate profile used for issuance. |
| endEntityProfile | text | End-entity profile used for issuance. |
| authType | select | mtls, bearer or basic. |
| username / usernamePrefix | text | API identity and optional prefix for created end entities. |
| clientCert, clientKey, trustBundle, token, password | secret | Sealed as one document. The trust bundle should contain the HTTPS server's root and the issuing CA whose chain you want stored. |
Required EJBCA role rules
The API identity needs at least: /ca/<issuing-CA>/, /ca_functionality/create_certificate/, /ca_functionality/view_ca/, /ca_functionality/view_certificate/, /ca_functionality/view_certificate_profiles/, /ra_functionality/create_end_entity/, /ra_functionality/view_end_entity/, /ra_functionality/revoke_end_entity/, and the matching create_end_entity, view_end_entity and revoke_end_entity rules for the chosen end-entity profile.
Lifecycle behaviour
- Test: authenticates to /ejbca/ejbca-rest-api/v1/ca and confirms the configured CA name is present.
- Issue: generates key and CSR locally, posts to /certificate/pkcs10enroll, then validates the returned certificate.
- Renew: same path, with the retained key (reuseKey) or a fresh key; the previous certificate becomes replaced.
- Revoke: parses the real issuer DN and calls PUT /certificate/<issuerDN>/<serial>/revoke?reason=…. Inventory changes only after EJBCA accepts.
- Templates: discovery reads the Certificate Profiles admin page with the connector's own credential and stores profiles of type End Entity.
Lab verification
cd rust-backend
python3 ../scripts/verify-ejbca-connector-e2e.sh
In the test profile the wizard already generated a paste-ready configuration at deploy/test-ejbca/connector-paste-values.txt. Treat that identity as disposable.
4.3HashiCorp Vault PKI connector
Signs a locally generated CSR through a Vault PKI role, verifies the returned chain, and revokes through the PKI engine. Lab-verified against Vault 1.18.5.
Configuration
| Field | Type | Notes |
|---|---|---|
| baseUrl | URL | Vault HTTPS origin. Plain HTTP is accepted only for a loopback development server. |
| mount | text | PKI secrets engine mount, default pki. |
| role | text | Signing role. A certificate profile can override it per request. |
| namespace | text | Optional X-Vault-Namespace. |
| authType | select | token or approle. |
| token | secret | Required for token auth. |
| roleId / secretId | secret | Required for AppRole auth. |
| caPem | secret | TLS trust anchor for the Vault listener. Not the PKI CA. |
| pkiCaPem | secret | Trust anchor for issued certificates. When empty, the issuing CA returned by Vault is used. |
Minimum Vault setup
vault secrets enable pki
vault write pki/root/generate/internal common_name="Lab Issuing CA" ttl=87600h
vault write pki/roles/spiryra allowed_domains="example.com" \
allow_subdomains=true max_ttl=720h key_type=ec
Grant the Spiryra identity read access to <mount>/ca/pem and <mount>/roles/<role> and update access to <mount>/sign/<role> and <mount>/revoke.
Lifecycle behaviour
- Test: resolves a token (or performs an AppRole login), reads the CA, and reads the role to prove it exists and report its allowed domains.
- Issue: POST /v1/<mount>/sign/<role> with the CSR, CN, DNS/IP SANs, requested TTL and format=pem.
- Renew: same path with a new key by default; reuseKey: true when a retained key exists.
- Revoke: POST /v1/<mount>/revoke; the serial appears in the Vault CRL.
4.4ACME v2 connector (HTTP-01, DNS-01, EAB)
Full ACME v2 lifecycle: account registration with a locally generated P-256 account key, order, authorization, finalize, download, chain verification, renewal and revocation.
Configuration
| Field | Type | Notes |
|---|---|---|
| directoryUrl | URL | ACME directory endpoint. HTTPS; loopback HTTP is accepted only for a local test server. |
| contactEmail | text | Optional account contact. |
| profile | text | Optional CA profile, for example Let's Encrypt tlsserver. |
| eabKid / eabHmacKey | text / secret | External Account Binding. Required by some providers; presets do not bypass it. |
| caPem | secret | TLS trust anchor for a private ACME server. Not the issuing CA. |
| dnsZoneId, dnsZoneName, dnsApiToken | text, text, secret | Set all three to enable DNS-01 for every order on this connector. Identifiers outside the zone are refused. |
| dnsApiBase | URL | Advanced: Cloudflare-compatible API base. |
| webroot | text | Optional absolute HTTP-01 challenge directory; defaults to SPIRYRA_ACME_WEBROOT. |
Presets are provided for Let's Encrypt staging and production, ZeroSSL, Google Trust Services and SSL.com. A preset fills the directory URL and a known profile only.
HTTP-01
Used when no DNS zone is configured. Spiryra writes <token>.<account-key-thumbprint> to $SPIRYRA_ACME_WEBROOT/.well-known/acme-challenge/<token> and removes it when the authorization completes. nginx serves that path from the same shared volume.
- Every requested name must resolve to this Spiryra deployment.
- The public proxy must forward port 80 requests for /.well-known/acme-challenge/ to the web container.
- Wildcards require DNS-01; a wildcard order on an HTTP-01-only connector is refused with an explicit error.
DNS-01
With a DNS zone configured, Spiryra creates a TXT record for each identifier, waits for propagation, completes the challenge, then removes the record. Cleanup failures are reported rather than hidden, and a journal entry is kept. If the API is killed mid-order, startup recovery removes the orphaned record after restart.
python3 scripts/verify-connectors-e2e.py --api-binary target/debug/spiryra-api exercises a full lifecycle against a local ACME stub, including wildcard issuance, a rejected TXT value, a rejected cleanup and an API kill during propagation.4.5Microsoft AD CS connector
Spiryra talks to AD CS without installing anything on the CA. Two clientless transports are supported, and CA work is never queued to an enrolled agent.
| Transport | How it reaches AD CS | Installed on the CA |
|---|---|---|
| winrm | certreq/certutil executed on a registered WinRM server | Nothing from Spiryra; built-in WinRM only |
| web | AD CS Web Enrollment (/certsrv) over HTTPS | Nothing from Spiryra; the optional Windows Web Enrollment role |
Setup with WinRM
- Register the CA host as a WinRM server target with a sealed credential (see 6.6).
- Create the connector with transport winrm, select the registered target, the CA name and a certificate template.
- Press Test. The response names the CA it reached and lists the templates available on it.
- Issue a certificate and confirm it appears in inventory with the CA as issuer.
WinRM and Web Enrollment never receive the CA private key or a CA administrator credential from the console.
Setup with Web Enrollment
- Install the Web Enrollment role on the CA and confirm https://<ca>/certsrv answers.
- Create the connector with transport web, the Web Enrollment URL, the template and — optionally — basic credentials. Set caPem to trust the site certificate.
- Confirm the account is allowed to enroll against the template: a local account on a workgroup CA, a domain account on a domain CA.
- Press Test, then issue.
The transport posts the CSR to /certsrv/certfnsh.asp with Mode=newreq, the base64 DER request, CertAttrib=CertificateTemplate:<template>, TargetStoreFlags=0 and SaveCert=yes, then downloads the returned certificate. Absolute and cross-origin certificate links are rejected.
4.6EST connector
RFC 7030 enrolment over HTTPS with a locally generated key and CSR. EST is opt-in per deployment, because the API must be told which hosts it may contact.
Enable EST
- Add the EST host to SPIRYRA_EST_ALLOWED_HOSTS in the profile env (comma-separated). An empty value keeps EST unavailable.
- Place the EST server's trust anchor in deploy/ca/ — it is mounted read-only at /etc/spiryra/ca — and reference it by absolute path.
- Redeploy the profile so the API picks up the new environment.
- Create the EST connector with the enrolment URL and trust anchor, then press Test.
Operations
| Action | Route | Notes |
|---|---|---|
| Inspect | POST /api/connectors/est/inspect | Reads the CA certificates and the enrolment endpoint before creating a connector. |
| Enrol | POST /api/connectors/est/enroll | Sends the PKCS#10 request and stores the returned certificate after validation. |
4.7SCEP connector
RFC 8894 SCEP enrolment through the packaged CMS exchange, pinned to the CA you configure. SCEP is opt-in, like EST.
Enable SCEP
- Add the SCEP host to SPIRYRA_SCEP_ALLOWED_HOSTS (comma-separated).
- Place the SCEP CA certificate and any intermediate chain in deploy/ca/.
- Redeploy, then create the SCEP connector with the service URL and the challenge password, and press Test.
Operations
| Action | Route | Notes |
|---|---|---|
| Inspect | POST /api/connectors/scep/inspect | Retrieves the CA certificate and confirms the endpoint answers. |
| PKI operation | POST /api/connectors/scep/pki-operation | Runs the enrolment exchange for one common name. |
4.8CA policy, templates and testing
Template discovery
Where the authority exposes them, Spiryra can pull the certificate profile list and store it for the issue form and policy checks. The result is visible under Connectors and in Permissions → CA policy.
GET /api/connectors/{id}/templates
POST /api/connectors/{id}/templates/sync
GET /api/ca/templates
CA policy
CA policy records which templates and profiles a deployment may use, and what key and validity constraints apply. Administrators manage it under Permissions → CA policy:
GET /api/access/ca-policy
POST /api/access/ca-policy
PATCH /api/access/ca-policy/{id}
Testing discipline
- Test after any credential rotation, CA upgrade or network change.
- Keep a connector that issues only to a non-production template for smoke tests, then delete that connector when it is no longer needed.
- Treat a CA-side rejection as an authority policy decision: read the returned message before changing anything in Spiryra.
5.1Import certificates
Importing brings an existing certificate and its chain into inventory so expiry tracking, ownership, deployment and reporting cover it.
- Open Issue & Import and choose Import.
- Paste or upload the PEM certificate. Add the chain when you want a full-chain bundle for deployment.
- Fill the operational metadata: owner team, application, business and technical contact.
- Save and confirm the parsed subject, SANs, key algorithm, issuer and expiry look correct.
POST /api/certificates/import
What import does and does not do
- It stores the certificate, chain and metadata; it does not obtain a private key.
- An imported certificate can be deployed to targets as a file — useful when the key is generated and held elsewhere.
- Rows that came from discovery or a legacy source may be metadata-only; policy findings only assess fields that were actually recorded.
5.2Issue certificates
Issuance always goes through a configured connector. Spiryra generates the private key locally, builds the PKCS#10 request, and validates the returned certificate before it enters inventory.
Issue from the console
- Choose the CA: pick a connector from the issuer list.
- Enter the common name and the SANs (DNS and IP). The SAN policy in Settings decides what the console allows.
- Choose key algorithm and size, requested validity and whether the key is retained in the vault.
- Optionally reference an existing deployment workflow. This screen attaches workflows that already exist in Orchestration and never creates one; the workflow's steps decide which files are written (leaf, chain, full-chain bundle).
- Submit and confirm the certificate in inventory.
POST /api/certificates/issue
Issue from your own CSR
If the key must never leave the requesting system, choose I have my own CSR and paste a PKCS#10 request. The CSR is validated (signature, key size, subject and SAN agreement) before it is sent to the CA.
POST /api/certificates/csr/sign
5.3Renew certificates
Renew on demand
- Open the certificate and choose Renew.
- Keep the same common name, SANs and issuer; decide whether to reuse the retained key or generate a new one.
- Submit. The new certificate is stored first; the previous one is then marked replaced.
POST /api/certificates/{id}/renew
POST /api/renewals/{certId}/retry
Scheduled renewal
The renewal scheduler opens work when a certificate enters the configured window, retries failures with backoff, and escalates a repeated-failure streak at most once per streak. Administrators can trigger a sweep:
GET /api/renewals
POST /api/renewals/run
Renewal plus deployment
A renewal can carry its deployment forward: the referenced workflow runs again — target, destination path, approved service reload and TLS check — and the deployment becomes verified only after the file readback and the presented fingerprint agree.
5.4Revoke certificates
- Open the certificate and choose Revoke.
- Choose the reason. The dialog repeats the subject and serial so you can confirm you are revoking the right certificate.
- Confirm. Spiryra asks the CA to revoke and updates inventory only after the CA accepts.
POST /api/certificates/{id}/revoke
What happens next
- The record is marked revoked with the reason and the CA's confirmation.
- If a retained key exists for that certificate, it is destroyed.
- The first revoked result seen by the revocation worker also writes a certificate.ca_revoked audit event and queues one ca_revocation notification.
- A CA-side revocation also reconciles inventory: the certificate stops reading as valid and is marked revoked once, with a certificate.ca_revocation_reconciled audit event. Only the state changes — the retained key is kept so the revocation can be investigated (an explicit revoke afterwards still destroys it), a certificate that was already replaced keeps that history, and replacing the certificate or rolling back a deployment stays an operator decision.
5.5Retained keys and downloads
Key retention
When retention is enabled at issue or import, the private key is written into the vault under the certificate id, sealed with AES-256-GCM using SPIRYRA_VAULT_KEY. Retention is what makes reuseKey renewal possible.
Downloads
| Part | Route | Control |
|---|---|---|
| Certificate | GET /api/certificates/{id}/download | certificates.export_public |
| Chain / bundle | GET /api/certificates/{id}/download?part=chain | certificates.export_public |
| Private key | GET /api/certificates/{id}/download?part=key | certificates.export_private plus an administrator session check in the handler |
Rotation and rewrap
Rotating the vault key rewraps retained leaf keys along with connector credentials and other sealed values (11.3). If a value cannot be opened, nothing is changed and the response names the affected table.
5.6Inventory, owners and expiry
What inventory holds
- Common name, SANs, key algorithm and size, issuer and issuer kind.
- Status: active, expiring, expired, revoked or replaced, with days remaining.
- Ownership: owner team, application, business and technical contact.
- Source: managed (issued through a connector), imported, or discovered from a target.
- Linkage to deployments and to orchestration runs that touched it.
Working with the list
- Filter by status, issuer, owner and expiry window; the expiring view includes already-expired certificates and those inside 90 days.
- Open a row for the chain, retained-key state, deployment history and audit trail.
- Assign or change the owner team where you have certificates.manage_owners (administrator-only by design).
- Export the current view for reporting; expiry and ownership reports page through the full record set.
Scoped visibility
An account with assigned scope only sees certificates it holds a grant for, and never sees another team's rows. Owners always see their own certificates.
5.7Certificate policy settings
Policy settings are what make the console refuse a request that would produce a certificate you cannot manage. Review them before the first issue.
| Setting | Effect |
|---|---|
| Expiry warning window | Days before expiry at which a certificate becomes expiring and starts appearing in alerts. |
| Expiry critical window | Days before expiry at which severity escalates and escalation notifications fire. |
| Default validity | Pre-filled lifetime for new requests, subject to the CA's own limits. |
| Allowed key algorithms and sizes | Restricts what the issue form and CSR signing accept. |
| SAN policy | Which names may appear in a request; wildcards where the CA supports them. |
| Session length | Console session lifetime (sessionHours). |
| Automatic renewal | Whether the scheduler opens renewals inside the window, and the retry policy. |
GET /api/settings
PATCH /api/settings
GET /api/selftest
6.1Deployment paths and the target model
There are three ways to deliver a certificate to a server. All three write the same artifact set, record the same evidence and refuse to claim success without reading the file back.
| Path | Transport | Best for |
|---|---|---|
| Native agent | Agent polls the control plane | Fleets you manage; no inbound firewall rule; local allowlists decide what may run. |
| Clientless SSH | API connects out on TCP 22 | Linux hosts where installing software is not allowed. |
| Clientless WinRM | API connects out on 5985/5986 | Windows hosts, including AD CS certificate stores. |
Objects
- Server — a registered host: an enrolled agent, an SSH target or a WinRM target, with its sealed credential.
- Installation (deployment target) — a certificate bound to a server and a destination: absolute path, optional service, optional TLS endpoint/SNI, and a 1–30 second probe timeout.
- Deployment attempt — one dispatch. It stores the expected SHA-256, queues the job, and only becomes active when the host reads the file back and reports a matching fingerprint.
Register a target
- Open Servers and choose the transport.
- For an agent, enroll it (6.2) and wait for its first poll. For SSH or WinRM, enter the host, port, user and credential — the credential is sealed immediately.
- Confirm the target shows ready: the agent is online, or the SSH/WinRM connection test passed.
Edit a target later
Service, TLS endpoint, SNI and timeout can be changed without detaching the target; the agent, destination, verified certificate and deployment history stay intact.
POST /api/installations
PATCH /api/installations/{id}
POST /api/installations/{id}/deploy
POST /api/installations/{id}/rollback
6.2Install and enroll the native agent
The agent is a standalone Go executable. It needs no Node.js, npm or JavaScript runtime, and it never listens on a port.
Download
Open Agent Downloads in the console and pick the platform, or fetch directly:
/downloads/agents/spiryra-agent-linux-amd64
/downloads/agents/spiryra-agent-linux-arm64
/downloads/agents/spiryra-agent-windows-amd64.exe
These paths serve allowlisted filenames without authentication, so a host can fetch its own binary before it holds a credential. Checksum or sign the file in your own pipeline if you need supply-chain proof.
Enroll one host
- Open Agent Fleet & Enrollment → Enroll agent. The console returns a one-time credential, shown once.
- Write it to a restricted file on the host (mode 0600, or an ACL-limited path on Windows).
- Set the environment and start the agent.
# Linux
export SPIRYRA_URL='https://spiryra.example.com'
export SPIRYRA_TOKEN_FILE='$HOME/.config/spiryra/agent-token'
export SPIRYRA_RESULT_OUTBOX_DIR='$HOME/.config/spiryra/result-outbox'
export SPIRYRA_ENROLLMENT_TOKEN_FILE='/secure/path/to/bootstrap-token'
chmod +x ./spiryra-agent-linux-amd64
./spiryra-agent-linux-amd64
# Windows (PowerShell)
$env:SPIRYRA_URL='https://spiryra.example.com'
$env:SPIRYRA_TOKEN_FILE="$env:ProgramData\Spiryra\agent-token"
$env:SPIRYRA_RESULT_OUTBOX_DIR="$env:ProgramData\Spiryra\result-outbox"
$env:SPIRYRA_ENROLLMENT_TOKEN_FILE='C:\secure\bootstrap-token'
.\spiryra-agent-windows-amd64.exe
The Windows .exe is the agent runtime; PowerShell only launches or installs it.
Bulk rollout with a campaign token
Instead of a per-host credential, set SPIRYRA_AGENT_ENROLLMENT_TOKEN (at least 32 characters) on the control plane and point each host's SPIRYRA_ENROLLMENT_TOKEN_FILE at that campaign secret. The host self-enrolls and receives its own identity. A campaign token is a shared secret for the duration of the rollout — rotate it afterwards.
A campaign token carries no principal, so on a deployment that hosts more than one tenant set SPIRYRA_AGENT_CAMPAIGN_TENANT to the tenant the campaign belongs to. Every credential it hands out, and therefore every finding those hosts report, is filed there; leave it empty to use the default tenant, which is correct for a single-tenant installation. A value that is not a tenant identifier refuses enrollment instead of falling back, so a typo cannot put a fleet's evidence into an estate that does not own those hosts.
Credential lifecycle
- The agent persists its credential, commits it, then polls /api/agent/jobs?version=… every 15 seconds.
- Every SPIRYRA_ROTATE_DAYS (default 30) it prepares a new token, persists it, then commits. Both tokens stay valid until commit finishes, so a crash cannot lock the host out.
- Removing an agent revokes its credential immediately and clears any pending rotation.
Choose the agent's capabilities
Enrollment asks which job kinds this host may receive. A credential is only handed the jobs its capabilities cover; a queued job for anything else is failed with a denial instead of being delivered later.
| Capability | Delivers |
|---|---|
| discovery | Local certificate and Windows store discovery. |
| deploy | Certificate file writes, the approved service reload and TLS evidence. |
| workflow | Approved orchestration host steps (copy, command, reload, verify). |
| windows-pfx | PFX import into the machine store and the optional IIS rebind. |
| tls-monitor | Monitored endpoint checks reported back to the console. |
| upgrade | Agent upgrade jobs and upgrade outcome reports. |
Change them later from Agent Fleet & Enrollment → Scopes (PATCH /api/agents/{id}/scopes). Narrowing applies to the next poll, and a job that was already dispatched is refused when the capability it needs has been removed. New enrollments start with every capability, so an existing host keeps working until you narrow it.
Optional: bind the credential to a client certificate
By default the token is the only thing an agent presents, so anyone who copies the file can poll that host's feed. Bind the credential to one client certificate and the token alone stops working: the request has to arrive through a TLS terminator that verified that exact certificate, and a different certificate is refused. The binding is per agent and optional, so an existing fleet keeps working until you enable it host by host.
- Terminate TLS in front of the API with client-certificate verification enabled. The shipped nginx configuration uses ssl_verify_client optional; the API speaks plain HTTP behind it and never sees the certificate itself.
- Forward what the terminator verified as x-client-cert-pem (nginx $ssl_client_escaped_cert), x-client-cert-fingerprint (SHA-256 of the DER, what Envoy's %DOWNSTREAM_PEER_FINGERPRINT_256% produces), or both, plus x-client-cert-verify with SUCCESS. The terminator must overwrite those headers; nginx does for the ones it sets.
- Bind the certificate when you enroll the host (mtlsFingerprint) or later with PATCH /api/agents/{id}/mtls. Upper case and colon separators are normalised to the stored lower-case digest.
- If the API is reachable without going through that terminator, set SPIRYRA_AGENT_MTLS_PROXY_SECRET on the control plane and have the terminator send it as x-spiryra-proxy-secret. The certificate headers are then ignored unless the secret matches, so a direct client cannot forge them.
- DELETE /api/agents/{id}/mtls clears the binding, and the same token works again with no certificate.
6.3Agent jobs, readback, reload and TLS verification
A deployment is a small pipeline on the host: write the file, read it back, reload the service, then probe what the socket actually presents. Each stage is recorded.
What the Linux agent needs before it installs anything
Certificate-file installation is gated twice on the host, and both gates are off by default. Until the lab opt-in and a usable recovery key are in place, an install job fails with a named error; the agent does not fall back to writing the file.
| Requirement | What happens without it |
|---|---|
| SPIRYRA_LAB_REMOTE_CERT_INSTALL=1 | The explicit opt-in. Unset, every install job fails with remote certificate installation is disabled. |
| SPIRYRA_INSTALL_ROOTS, SPIRYRA_RECOVERY_JOURNAL_DIR and SPIRYRA_RECOVERY_KEY_FILE | All three are needed together. If any is unset the job fails with agent installation allowlist and recovery configuration required. |
| A recovery key of exactly 32 raw bytes, kept at mode 0600 | The contents are checked, not the mode. A missing or short key fails the job with valid 32-byte recovery key unavailable, and batch rollback refuses to run, rather than write a file that cannot be undone. |
The recovery key is a durable host secret, not a per-run value: the journal sealed with it before the write must still be readable after a restart. Generate one with openssl rand 32, keep it outside the certificate directories, and back it up separately — a lost key leaves an interrupted deployment with no rollback path. Set SPIRYRA_RECOVER_ON_STARTUP=1 to replay an unfinished journal at boot; the agent then refuses to start, instead of serving jobs, when the key or the journal directory is missing.
The script the agent enforces
- Write the certificate material to the destination path, inside SPIRYRA_INSTALL_ROOTS. A path outside those roots is refused.
- Read back the file and report its SHA-256. The deployment fails if it differs from the expected fingerprint.
- Reload the named service, but only when it appears in SPIRYRA_APPROVED_SERVICES. A service outside the list is not touched.
- Probe the TLS endpoint, only if it appears in SPIRYRA_TLS_PROBE_ALLOWLIST; an empty list denies every probe. The requested timeout is capped by SPIRYRA_TLS_PROBE_MAX_TIMEOUT_SECS before any file is written.
- Report the leaf certificate actually served — fingerprint, subject and expiry — plus the effective timeout and reload result.
What the control plane requires
- A timeout and a matching presented fingerprint before a deployment is marked verified.
- Both the requested and the effective timeout are stored, and audit events carry the effective value.
- A mismatch fails closed and queues an automatic rollback when the key did not change.
- Results are delivered from a durable local outbox, so a brief network failure does not lose evidence.
A deployment is a workflow you build and test in Orchestration, then reference from the issue or import screen. Those screens only attach an existing workflow — they never generate one — so the artifact list, reload and verification steps are the same for a first issue and for every renewal.
6.4Agent fleet and signed upgrades
Fleet view
GET /api/agents
GET /api/agents/fleet
POST /api/agents
POST /api/agents/bulk
DELETE /api/agents/{id}
The list shows total, online, outdated and never-connected counts, each agent's build label and last poll, and the last 25 jobs. Bulk enrollment creates one identity and one token per host — revoking one agent never disturbs the rest.
How an upgrade works
- Publish a release entry in SPIRYRA_AGENT_RELEASE_CATALOG as local JSON: { "releases": [ { "version", "os", "arch", "sha256", "url", "signature" } ] }.
- The agent stages a release only when SPIRYRA_AGENT_UPGRADES_ENABLED=1, a pinned SPIRYRA_UPDATE_SIGNING_PUBLIC_KEY exists, and the stage and intent directories are absolute paths.
- The Linux package already installs the supervisor, its timer and a root-owned /etc/spiryra-agent/supervisor.env; fill in that file and enable spiryra-upgrade-supervisor.timer. For a hand-managed install use scripts/install-upgrade-supervisor-linux.sh (root) or scripts/install-upgrade-supervisor-windows.ps1 (elevated). Set SPIRYRA_AGENT_EXECUTABLE to the installed agent binary, never to the supervisor.
- Create a rollout, dispatch it, and watch outcomes. Agent completion means staged, not installed.
POST /api/agent-upgrades/rollouts
POST /api/agent-upgrades/rollouts/{id}/approve
POST /api/agent-upgrades/rollouts/{id}/dispatch
GET /api/agent-upgrades/rollouts/{id}/outcomes
POST /api/agent-upgrades/rollouts/{id}/cancel
Installing the agent
scripts/package-agent.sh builds the installable artifacts from one source tree:
- spiryra-agent_<version>_amd64.deb and _arm64.deb for Debian and Ubuntu.
- spiryra-agent-<version>-1.x86_64.rpm and .aarch64.rpm for RHEL, Rocky, Alma and SUSE.
- spiryra-agent-<version>-windows-amd64.zip: the executable, both installer scripts and INSTALL.txt.
- SHA256SUMS, plus SHA256SUMS.sig when SPIRYRA_PACKAGE_SIGNING_KEY names a key.
A package installs the agent at /usr/local/bin/spiryra-agent, a separate supervisor copy at /usr/local/libexec/spiryra-upgrade-supervisor, the three systemd units, and 0600 root-owned configuration. It does not start anything: the agent has no control-plane URL or credential until you set SPIRYRA_URL and enable spiryra-agent.service.
The gate's agent-packages stage builds these artifacts and installs them in throwaway Debian and EL9 containers, so a package that installs with the wrong file mode, lands a unit where the distribution does not read it, or fails to remove cleanly fails the release.
6.5Clientless SSH delivery
No agent is installed. The control plane connects with a sealed SSH key, writes the public certificate, reloads an approved service and verifies the file and the TLS endpoint.
Prerequisites on the target
- SSH reachable from the API container on TCP 22.
- A user with write access to the certificate path and permission to reload the service.
- A host key recorded when you register the server — the connection is pinned to it.
Register and deploy
- Servers → Add server → SSH. Enter host, port and user, then paste the private key (and an OpenSSH user certificate if your estate requires one).
- Save. The credential is sealed with the vault key and never returned.
- Create the deployment target: absolute certificate path, optional file mode, optional service reload, optional TLS endpoint.
- Deploy. The API writes the file, reloads the approved service, reads the file back and probes the endpoint.
Safety properties
- The command executed for a reload is fixed by the target configuration, not composed from user input.
- A failed write leaves the previous file in place; the deployment records the failure instead of retrying silently.
- Rollback reinstalls the previous certificate with a fresh verification and marks the original deployment restored.
6.6Clientless WinRM delivery
Windows targets are reached over WinRM with a sealed password. Certificate files are written, services restarted, and the presented certificate reported — all without installing Spiryra on the host.
Prerequisites
- WinRM enabled. Production should use HTTPS on 5986 with a listener certificate the API trusts.
- Basic authentication enabled on the listener. Windows disables it by default and refuses every Spiryra request with WinRM authentication failed until it is on. In an elevated PowerShell on the target:
Set-Item WSMan:\localhost\Service\Auth\Basic -Value $true
For a lab listener on 5985, unencrypted traffic must be allowed as well: Set-Item WSMan:\localhost\Service\AllowUnencrypted -Value $true and Restart-Service WinRM. Do not enable AllowUnencrypted on a host reachable outside its management network; use 5986 there. - An account that can write the destination path and restart the named service.
- SPIRYRA_AGENTLESS_WINRM_ALLOW_HTTP=true only for a lab using 5985.
Register and deploy
- Servers → Add server → WinRM: host, port, user and password (sealed immediately).
- Create the deployment target with the Windows path, the service to restart and the TLS endpoint to probe.
- Deploy. The API writes the file, restarts the service, then reads the certificate the socket serves and compares its fingerprint.
Script behaviour
| Action | Success | Failure |
|---|---|---|
| Write certificate | File digest matches | Non-zero exit; deployment fails |
| Restart service | reload=succeeded | Unknown service fails with a legible error |
| Probe endpoint | Fingerprint matches, expiry recorded, no error line | Closed port or mismatch fails the check |
6.7Windows PFX store, service-account ACL and IIS
Some Windows services need the certificate in the machine store rather than a file. The agent imports the PFX, binds the private key to the service identity that must read it, and can move — or create — an IIS HTTPS binding for the imported certificate in the same job.
How the import works
- Stage the PFX and a DPAPI-protected password file locally on the target. A user-scope blob (ConvertFrom-SecureString) works when the staging account also imports; a machine-scope blob is accepted as well, and is what you need when a service account, WinRM or SYSTEM does the import. The API payload has no password field.
- The agent imports into LocalMachine/My with the machine key set; the recorded evidence includes the store, hasPrivateKey and the key method (cng or capi).
- Set SPIRYRA_WINDOWS_PFX_SERVICE_ACCOUNT to the SAM or UPN identity that reads the key. A bare local name is machine-qualified before use.
- The agent writes an Allow-Read ACE on the key container (Microsoft\Crypto\Keys for CNG, Microsoft\Crypto\RSA\MachineKeys for CAPI), reads it back, and fails the job if it could not be written.
Set SPIRYRA_WINDOWS_PFX_REQUIRE_SERVICE_ACCOUNT=true to fail closed when no identity is configured. The account is host configuration only, so a control-plane caller cannot choose which identity receives private-key access. The control plane also rejects a succeeded result whose ACL evidence is partial.
IIS certificate changeout
IIS serves TLS through HTTP.sys, which accepts only a certificate that is in LocalMachine\My with its private key. A file deployment of the public PEM therefore cannot renew an IIS binding; the rebind belongs to the PFX import that put the key there.
- Open Installations → Windows IIS changeout and choose the channel: a Windows agent that already holds the staged PFX, or a registered WinRM server with no agent.
- Name the staged PFX and DPAPI password files and give the certificate's public SHA-256 fingerprint.
- Set the IIS site and its HTTPS binding, for example Default Web Site and *:443:www.example.test. Supply both or neither.
- The transport imports the certificate, then matches the site's HTTPS bindings: exactly one match is moved to the new thumbprint with the previous one recorded, and a site that has none gets the binding created for this job. The HTTP.sys registration is read back before success is reported.
- Optionally name an application pool (agent channel only) to have the job recycle it once the binding is confirmed, so the worker that has to pick up the new key restarts without you. It runs after the bind, outside the rollback path: a failed restart reports the reason and leaves the certificate serving.
- The row appears under Windows store imports with the store, private-key ACL, IIS site/binding, requested pool and recycle evidence, previous and applied thumbprints, or the rejection reason.
Agent or WinRM
| Channel | Where the PFX is staged | Result |
|---|---|---|
| Spiryra agent | Under SPIRYRA_WINDOWS_PFX_STAGING_DIR on the agent host, which needs SPIRYRA_WINDOWS_PFX_ENABLED=true. | Queued (202); the agent reports store, ACL and IIS evidence on its next poll. |
| WinRM (no agent) | Under SPIRYRA_WINRM_PFX_STAGING_DIR on the target. The control plane needs SPIRYRA_AGENTLESS_ENABLED=true and the host in SPIRYRA_AGENTLESS_WINRM_HOSTS. | Synchronous; success returns the evidence, a rejection returns 502 and records the remote reason in the history. |
Binding the key to an identity grants use of it, not only an entry in a list. On a live Windows Server 2025 host the agent's import binds NT AUTHORITY\NETWORK SERVICE with the same Set-Acl call it uses in production, and a probe running as that identity signs a payload with the key; an identity without the ACE is refused, and the control leg binds that second identity and has it sign, which shows the refusal came from the grant. Run it with scripts/verify-windows-pfx-identity-e2e.py; it removes the certificate, the probe and any task it created.
A script that does not fit one WS-Man command line is uploaded to the target's temp directory in bounded chunks and run from that file. Sent inline, WS-Man answers The command line is too long and runs nothing, so the control plane never sends the import inline.
POST /api/installations/windows-pfx
{"id":"job_pfx_000001","agentId":"agent_00001",
"pfxName":"tls-server.pfx","passwordName":"tls-server.dpapi",
"expectedSha256":"<64 hex digits>",
"iisSite":"Default Web Site","iisBinding":"*:443:www.example.test"}
# after a confirmed bind, recycle every pool that has to load the new key
{"id":"job_pfx_000002","agentId":"agent_00001",
"pfxName":"tls-server.pfx","passwordName":"tls-server.dpapi",
"expectedSha256":"<64 hex digits>",
"iisSite":"Default Web Site","iisBinding":"*:443:www.example.test",
"iisAppPools":["DefaultAppPool","Reports Pool"]}
Binding rules and evidence
| Input | Accepted | Refused |
|---|---|---|
| Site | Letters, digits, spaces, ._-; 1–64 characters | Quoting or shell characters, leading or trailing spaces |
| Binding | ip:port:host with * or IPv4, port 1–65535, empty/DNS/leading-wildcard host | IPv6 literals, port zero, unknown scheme shapes |
| HTTPS binding | Exactly one matching HTTPS binding is moved. The agent channel creates one when the site has none; a WinRM import only moves a binding that already exists | More than one match, or a site that does not exist; the job fails without changing IIS |
| Application pools | Up to eight pool names in iisAppPools (iisAppPool for one), letters, digits, spaces, ._-, 1–64 characters each, with a site and binding; each reused after the bind is confirmed | A pool list without a site and binding; an unknown pool; a duplicate name; a pool nobody asked for in the result |
| Evidence | applied=true, matching site, binding and thumbprint, created, previousThumbprint (null when the binding was created), and appPools naming each recycled pool (plus appPool for a single one) | Missing or partial evidence; a rebind or a recycle reported for a job that did not ask for one; one recycled pool standing in for the rest |
6.8Deployment rollback and recovery
Single deployment rollback
- Open the certificate's deployment history and choose Rollback on the active deployment.
- The previous certificate is reinstalled, read back and verified.
- The original deployment is marked restored and the rollback is recorded with its own evidence.
POST /api/installations/{id}/rollback
When a write succeeds but verification fails
If the file matches but the presented fingerprint does not, the deployment fails and — when the key did not change — an automatic rollback is queued. If the key did change, the old certificate cannot restore the service, so the failure is surfaced for an operator instead of pretending to fix it.
Recovery ledger
Actions with no automatic inverse stay visible: run commands and external API calls in a batch rollback are written to a recovery ledger with the command or call to review. The deployment detail lists what was reloaded, what needs manual action and who should review it.
7.1Orchestration concepts
Orchestration turns a repeatable certificate operation into a durable, auditable run: write files, reload services, verify endpoints, wait, ask for approval, call an API.
| Object | Meaning |
|---|---|
| Workflow | The definition: ordered steps bound to a certificate and one or more targets. |
| Step | One action. Host steps run on the target; control-plane steps run in the API. |
| Run | One execution against one target, with per-step state, output and evidence. |
| Batch | One run across several targets, created atomically. |
| Approval | A gate that stops a run until an administrator decides. |
Step kinds
| Kind | Runs where | What it does |
|---|---|---|
| copy_file | Host | Writes public certificate material inside the agent's SPIRYRA_INSTALL_ROOTS. |
| run_command | Host | Runs only a command that exactly matches SPIRYRA_APPROVED_COMMANDS on the agent. |
| reload_service | Host | Reloads or restarts a service in SPIRYRA_APPROVED_SERVICES. |
| verify | Host | Probes an endpoint in SPIRYRA_TLS_PROBE_ALLOWLIST and reports the certificate presented. |
| discover | Host | Inventories certificate files and records findings. |
| windows_pfx_import | Host (WinRM only) | Imports a PFX the operator staged on the server into LocalMachine\My and can move one IIS HTTPS binding. It calls the same import as the certificate page, so the fingerprint check, the store readback and the history row are identical. Agent and SSH targets are refused: the agent's import runs as a queued job with its own service-account ACL. |
| api_call | Control plane | HTTPS request for SaaS uploads and health checks. |
| send_email | Control plane | Emails one or more recipients. Needs SMTP under Settings → Notifications; use Send email notification in the palette. |
| notify | Control plane | Queues email or a configured Slack/Teams/PagerDuty/generic webhook. |
| delay | Control plane | Bounded wait before the next step. |
| approval_gate | Control plane | Stops until an administrator approves or denies. |
Placeholders
Steps can reference the run's certificate and target: {{cert.pem}}, {{cert.pemEscaped}}, {{cert.chain}}, {{cert.fullchain}}, {{cert.commonName}}, {{cert.serial}}, {{cert.fingerprint}}, {{target.name}}, {{target.host}}, {{target.platform}}, {{target.fileStem}}, {{workflow.name}} and {{run.id}}.
GET /api/orchestration/meta
GET /api/workflows
GET /api/runs
GET /api/runs/{id}
7.2Build a workflow
- Open Orchestration → New workflow and name it for the operation, not the environment.
- Bind the certificate and the targets. A target must already be registered (6.1).
- Drag steps from the palette onto the canvas and order them. Host steps need an agent, SSH or WinRM transport that the target supports.
- Fill each step: destination path and file mode for copy_file; the approved service for reload_service; host, port and SNI for verify; URL, method and body for api_call.
- Save, then run against one target first. Batches come after a single run succeeds.
Design rules that keep runs debuggable
- Keep copy_file → reload_service → verify contiguous. That is the sequence that produces usable evidence.
- Put an approval_gate before anything that reaches a customer-facing service.
- Prefer several small workflows to one long chain: a failure then leaves a shorter, clearer state to reason about.
- Use delay only where a real convergence window exists, not as a substitute for verification.
7.3Host step safety and conditions
Allowlists decide, not the workflow
| Agent variable | Governs | Default |
|---|---|---|
| SPIRYRA_LAB_REMOTE_CERT_INSTALL | Whether the agent performs certificate-file installation at all. | Unset disables every install. |
| SPIRYRA_INSTALL_ROOTS | Directories a copy_file may write into. | Unset denies every write. |
| SPIRYRA_RECOVERY_KEY_FILE | The 32-byte key that seals the rollback journal. | Unset disables installs and batch rollback. |
| SPIRYRA_APPROVED_COMMANDS | Exact command strings a run_command may execute. | Unset denies every command. |
| SPIRYRA_APPROVED_SERVICES | Services a reload may touch. | Unset denies every reload. |
| SPIRYRA_TLS_PROBE_ALLOWLIST | Endpoints a verification may probe. | Empty denies every probe. |
| SPIRYRA_TLS_PROBE_MAX_TIMEOUT_SECS | Upper bound on a requested probe timeout. | Local cap wins over the request. |
A workflow that asks for something outside these lists fails at the host with a clear error. Widening an allowlist is a host-side change, made deliberately on that server — never a control-plane setting.
Allowlists for clientless SSH steps
When a step runs over SSH instead of through an agent, the same model is enforced from the API process, with its own lists so a server does not inherit another path's grants. These are API-host environment variables, and an empty list denies the operation.
| API variable | Governs | Default |
|---|---|---|
| SPIRYRA_SSH_INSTALL_ROOTS | Comma-separated directories a clientless copy_file may write into. | Unset denies every write. |
| SPIRYRA_SSH_APPROVED_COMMANDS | Exact command strings, one per line, a clientless run_command may execute. | Unset denies every command. |
| SPIRYRA_SSH_APPROVED_SERVICES | Comma-separated services a clientless reload may touch. | Unset denies every reload. |
| SPIRYRA_SSH_TLS_PROBE_ALLOWLIST | Comma-separated host:port endpoints a clientless verification may probe, resolved on the target. | Empty denies every probe. |
| SPIRYRA_AGENTLESS_SSH_KNOWN_HOSTS | The pinned host-key file. Required: without it no SSH step can open a session. | Unset blocks every SSH step. |
| SPIRYRA_AGENTLESS_SSH_HOSTS | Optional host allowlist shared with clientless deployment. | Empty means the registered server is the boundary. |
A clientless write accepts certificate PEM blocks only, verifies the digest after the move, and records backup: none — there is no sealed journal on the API side, so a failed batch lists the write for operator review instead of claiming it restored the file.
Allowlists for clientless WinRM steps
A server registered with the winrm transport runs the same four host steps from the API process, over the sealed password and the same HTTPS listener a clientless deployment uses. The lists are separate from SSH's, so a Windows fleet never inherits a Linux path grant. An empty list denies the operation.
| API variable | Governs | Default |
|---|---|---|
| SPIRYRA_AGENTLESS_WINRM_HOSTS | Comma-separated hosts WinRM may connect to. A server outside it is reported as not runnable. | Empty blocks every WinRM step. |
| SPIRYRA_WINRM_INSTALL_ROOTS | Comma-separated drive-absolute directories a clientless copy_file may write into, for example C:\ProgramData\Spiryra. Prefix matches respect the directory boundary. | Unset denies every write. |
| SPIRYRA_WINRM_APPROVED_COMMANDS | Exact PowerShell statements, one per line, a clientless run_command may execute. | Unset denies every command. |
| SPIRYRA_WINRM_APPROVED_SERVICES | Comma-separated services a clientless reload may restart. | Unset denies every reload. |
| SPIRYRA_WINRM_TLS_PROBE_ALLOWLIST | Comma-separated host:port endpoints a clientless verification may probe, read from the target. | Empty denies every probe. |
| SPIRYRA_AGENTLESS_WINRM_ALLOW_HTTP | Permits port 5985 over plain HTTP. Off by default: 5986 with certificate validation is the supported path. | Unset refuses HTTP. |
Windows has no service reload, so a reload_service step restarts the service and its evidence records mode: reload next to applied: restart — the run says what happened rather than implying a graceful reload. A WinRM write accepts certificate PEM blocks only, checks the digest before the move and after reading the file back, and also records backup: none.
Conditions
A step can carry a condition over the run's state, so a chain can branch on what actually happened: proceed only when the certificate is active, skip an API call when verification failed, or stop before a reload when the file digest already matches. Conditions are evaluated against recorded step output, not against assumptions.
7.4Approval gates
An approval_gate stops the run before the next step. The run waits in Orchestration → Approvals, where an administrator approves or denies it with a reason.
GET /api/orchestration/approvals
POST /api/runs/{id}/approval
| Decision | Effect |
|---|---|
| Approve | The run continues at the step after the gate; the approver, time and reason are recorded. |
| Deny | The run stops. Steps already completed stay completed; no further step runs. |
| Timeout | The gate expires: the run stops waiting rather than proceeding by default. |
7.5Parallel groups and multi-target batches
Parallel groups
A workflow may contain bounded, contiguous, read-only groups that run concurrently — for example discovery or verification across several endpoints. Write steps never run in parallel: a reload or a file write stays sequential so the evidence order is unambiguous.
Batches
- Batch creation is atomic: every target gets a run, or none does.
- A failed target halts its siblings before their next step, so a broken certificate does not spread across the estate.
- Approved service reloads inside a batch are re-issued after restores, so the service ends up running against the files that are actually on disk.
- Batch detail shows per-target state, plus what was restored, what was reloaded and what needs review.
POST /api/runs
GET /api/runs/{id}
POST /api/workflows/{id}/run-batch
7.6Batch rollback and the recovery ledger
A failed batch compensates every succeeded step, not just file writes. The plan is built in reverse completion order and each part is recorded.
- Restore files. Sealed per-step backups are restored first, so the host holds the previous certificate material.
- Re-issue reloads. Each approved service reload is issued again so the service runs against the restored files.
- Record the irreversible. Steps with no automatic inverse — run_command, api_call and any future kind — are written to the recovery ledger with the kind, reason and the command or call to review.
The batch detail carries reloaded, manual, manualSteps and operatorReview; the audit event stays batch.rolled_back at warning level while listing what an operator must finish.
Restart behaviour
On startup the API resumes queued and running runs, preserves succeeded steps, reattaches host steps to their existing agent job and fails in-flight control-plane steps closed. A database lease prevents two API replicas from driving the same run.
8.1Server inventory and findings
Discovery finds the certificates already on your servers, so they can be tracked, corrected or promoted into inventory — before they expire in a place nobody is watching.
Run a discovery
- Open Servers and choose a ready agent, SSH or WinRM target.
- Run discovery. The agent inventories certificate files inside its install roots; SSH and WinRM targets follow the same allowlisted paths.
- Review the findings: path, subject, issuer, expiry and the fields that could not be read.
- Promote a verified public certificate into inventory, or correct the finding state so it stops appearing as new.
GET /api/findings
PATCH /api/findings/{id}/state
POST /api/findings/{id}/import
GET /api/installations
What discovery does not do
- It does not read or exfiltrate private keys. Findings are certificate metadata.
- It does not scan arbitrary paths outside the agent's install roots.
- It does not merge duplicates silently — a promoted finding becomes a certificate row with its own source.
8.2TLS presentation monitoring
Deployment verification proves what a socket served at one moment. Continuous monitoring proves it kept serving the right certificate afterwards.
Create a monitor
- Open Monitoring → Add endpoint and enter host, port and — when the service needs it — SNI.
- Choose the expected certificate, or let the monitor record the first observed leaf as its baseline.
- Save and run a scan to confirm the endpoint is reachable from the API.
GET /api/monitor/tls
POST /api/monitor/tls
GET /api/monitor/tls/{id}/history
POST /api/monitor/scan
What each check records
| Signal | Why it matters |
|---|---|
| Presented fingerprint | Detects an unexpected certificate, a rolled-back file or a service that failed to reload. |
| Subject and SAN coverage | Detects a certificate that does not cover the name clients use. |
| Expiry | Drives alerts before the endpoint starts failing. |
| Chain and trust | Detects a missing intermediate, the most common cause of “works in the browser, fails in the client”. |
| Probe error | Refused connection, timeout or TLS handshake failure. |
Alerts are raised on change and on expiry thresholds, and can be acknowledged per endpoint. History is retained so you can answer “when did this change?” rather than only “is it wrong now?”.
8.3Revocation checks (OCSP and CRL)
For active certificates, the revocation worker polls the authority's OCSP responder or CRL endpoints and records what it finds. This closes the gap between “Spiryra thinks it is active” and “the CA says it was revoked”.
What happens on a revoked result
- The certificate's recorded revocation state is updated from the CA answer.
- On the first revoked result, a certificate.ca_revoked audit event is written.
- One ca_revocation notification is queued to the owner and configured channels — once per certificate, not on every poll.
- Inventory state is reconciled to revoked once, with a certificate.ca_revocation_reconciled event. Replacing the certificate and rolling back a deployment remain operator decisions, and the retained key is kept for investigation.
Status and limits
- A real Google Trust Services CRL was checked in the lab.
- OCSP runs against a real OpenSSL responder in the test suite (revocation::tests::a_live_responder_reports_good_and_revoked): one certificate it reports good and one revoked, through the same request path the worker uses. The responder is generated by the test, so no external service is needed.
- A CA that publishes neither OCSP nor CRL cannot be checked — record that as a known gap for that issuer.
8.4Alerts and acknowledgement
Alerts are derived from certificate expiry and monitoring observations, then acknowledged or resolved by operators. The alert row is the alert's own record, not a copy of the certificate: it keeps the last observation (severity, message, threshold, common name, expiry and days remaining, as seen at the last scan), the alert lifetime (createdAt, lastSeenAt) and the operator status. The list is the live expiry projection merged with those rows, so history survives a certificate leaving the warning window. A scan reopens an alert whose condition returns and reopens an acknowledged alert that becomes critical; a resolved alert stays readable under ?status=resolved.
GET /api/alerts
POST /api/alerts/{id}/ack
POST /api/alerts/{id}/resolve
| Severity | Typical trigger | Expected response |
|---|---|---|
| Critical | Expired, or inside the critical window, or a revoked certificate still served. | Replace the certificate now; confirm the endpoint after deployment. |
| Warning | Inside the warning window, or a presented fingerprint changed unexpectedly. | Schedule renewal; confirm the change was intended. |
| Info | Discovery found a new certificate, or a monitor was created. | Decide whether to bring it under management. |
9.1Local accounts and roles
Roles
| Role | Can read | Can write | Notes |
|---|---|---|---|
| Viewer | Own team's certificates and servers | No | Read-only; cannot issue, deploy or approve. |
| Operator | Own team's resources | Yes, within grants and policy | Issues, deploys and runs workflows; cannot change identity or settings. |
| Administrator | Everything | Everything | Identity, settings, connectors, approvals, service accounts. |
Manage accounts
GET /api/users
POST /api/users
PATCH /api/users/{id}
DELETE /api/users/{id}
Administrators create accounts, change roles, reset passwords, disable and delete. The last active administrator cannot be disabled or deleted — the console refuses rather than locking the deployment out.
Durability
- Accounts live in clm_users as Argon2id hashes; sessions live in clm_sessions as SHA-256 digests.
- Sessions survive an API restart until they expire.
- Disabling an account revokes its sessions immediately.
- The environment bootstrap administrator exists to create the first real accounts — it is not a shared operational login.
9.2Teams, ownership and grants
Roles say what kind of action you may take. Teams, ownership and grants say which resources you may take it on.
Three layers
- Role — viewer, operator or administrator.
- Scope — all (every resource) or assigned (only explicit grants). Existing installations default to all, so enabling grants does not silently remove access.
- Grant — one row binding an account to a certificate or connector with read or manage.
Owners always see and manage their own certificates, even on an assigned account. A write additionally requires an operator-or-higher role and a manage grant.
Configure access
- Open Permissions and create or select a team.
- Set the account's scope to assigned when it should be limited.
- Add grants: choose the certificate or connector and the permission level.
- Use Preview to ask what the account would be allowed to do before you save.
GET /api/access/summary
GET /api/access/catalog
POST /api/access/teams
POST /api/access/owners
GET /api/access/preview
GET /api/access/audit
What filtering actually does
- Inventory, detail, orders and connector reads are filtered by grant; a row is visible when its server or its certificate is granted.
- Writes outside a grant are refused with 403, not silently dropped.
- A hidden resource is indistinguishable from a missing one, so grants do not leak existence.
9.3OpenID Connect
OIDC sign-in uses the authorization-code flow with PKCE, nonce, state and return-path handling, and validates the ID token signature, issuer, audience, expiry and nonce before a session is created.
Configuration
| Field | Value |
|---|---|
| issuer | Provider issuer URL. Discovery and JWKS are read from it. |
| clientId | Client registered with the provider. |
| clientSecret | Secret. Sealed in the vault; never returned by the API. |
| redirectUri | Must be <public origin>/api/auth/oidc/callback and registered exactly at the provider. |
| scopes | Usually openid profile email. |
| label | Button text on the sign-in page. |
| defaultRole | Role for a first-time federated account. |
| tokenAuth | basic (client secret basic) or post (client secret post). |
Set it up
- Register the application at the provider with the exact redirect URI above.
- In Settings → Identity providers, enter the issuer and client values and save.
- Use the built-in Check action: discovery, JWKS reachability, issuer and client consistency are reported per check.
- Sign in through the provider in a private window and confirm the resulting role and team.
GET /api/auth/providers
GET /api/auth/oidc/login
GET /api/auth/oidc/callback
Signing in once a provider is configured
The provider becomes the default. Opening the console signed out redirects to it after a four-second hold, so somebody who only knows the site address never chooses a method. The hold is deliberate: the same page carries the escape hatch, and a redirect nobody can interrupt is a redirect that locks you out when the identity provider is down.
- Bypass SSO on the sign-in page stops the redirect and opens the local sign-in address.
- /login/local reaches the same screen directly, with no redirect. This is the address to bookmark and to keep in a runbook. GET /api/auth/providers returns it as localLoginUrl, and the console reads that value instead of hardcoding a path.
- Local (break-glass) accounts sign in with username and password, exactly as they did before federation was configured. Password expiry, the administrator recovery token and the expired-password change all work there.
- API key signs in as a scoped service account. Paste the token issued on Settings → Service Accounts; the console presents it as a bearer credential, so the session shows only the resource types and actions that token was scoped for, and never becomes an administrator session. Revoking the token ends the session as soon as the next request is made, and the key is held in the tab's session storage rather than a cookie.
9.4SAML 2.0
SAML sign-in uses SP-initiated AuthnRequests with signed assertions, replay protection and strict issuer matching. Once it is configured it is the default sign-in method; the break-glass address is /login/local (see 9.3).
Configuration
| Field | Value |
|---|---|
| entryPoint | Identity provider SSO URL (HTTP-Redirect binding) from its metadata. |
| idpIssuer | IdP entity ID. Assertions from any other issuer are rejected. |
| idpCert | Secret. IdP signing certificate as a PEM block or the base64 value from metadata. |
| spEntityId | Service provider entity ID. Leave blank to use the metadata URL. |
| acsUrl | Assertion consumer service: <public origin>/api/auth/saml/acs. |
| defaultRole | Role for a first-time federated account. |
| allowIdpInitiated | Whether unsolicited assertions are accepted. Off is safer. |
Exchange metadata
- Download /api/auth/saml/metadata and give it to the IdP administrator, or copy the entity ID and ACS URL.
- Map the IdP's user identifier to the account email used in Spiryra.
- Paste the IdP certificate and SSO URL here, then save.
- Sign in through the IdP in a private window and confirm the role.
GET /api/auth/saml/metadata
GET /api/auth/saml/login
POST /api/auth/saml/acs
9.5SCIM 2.0 provisioning
SCIM lets your identity provider create, update, deactivate and group users automatically. Spiryra exposes the standard Users and Groups resources plus discovery documents.
Endpoints
GET /scim/v2/ServiceProviderConfig
GET /scim/v2/ResourceTypes
GET /scim/v2/Users POST /scim/v2/Users
GET /scim/v2/Users/{id} PUT /scim/v2/Users/{id}
PATCH /scim/v2/Users/{id} DELETE /scim/v2/Users/{id}
GET /scim/v2/Groups POST /scim/v2/Groups
GET /scim/v2/Groups/{id} PUT /scim/v2/Groups/{id}
PATCH /scim/v2/Groups/{id} DELETE /scim/v2/Groups/{id}
Connect the provider
- In Settings → Identity providers → SCIM, generate a SCIM token. It is shown once.
- Configure the provider with the base URL https://spiryra.example.com/scim/v2 and that token.
- Map IdP groups to Spiryra roles or teams under Permissions.
- Provision one test user and confirm they appear with the expected role before enabling the full directory sync.
Fail-closed behaviour
- A SCIM-managed account is always evaluated against custom policies; its legacy scope never widens access.
- Removing the last group-to-role mapping narrows the account instead of leaving it with its previous role.
- Deactivating a user removes access immediately; deleting a user preserves the audit trail.
9.6API service accounts
Machine credentials for automation that should not sign in as a person. A token carries an explicit allow-list drawn from the console's permission catalogue. It can be a bearer secret, or a certificate credential with no secret at all.
Credential shape
Authorization: Bearer csa.<token id>.<secret>
The secret is 32 random bytes, base64url, 43 characters. Only its SHA-256 digest is stored, the plaintext is returned once at creation, and listing never returns the digest or the credential.
Certificate credential (no secret)
Issue a token with clientCertificate set to the PEM of the certificate the automation host will present, and no secret is created or returned: the response carries authMethod: "mtls" and the bound fingerprint instead. From then on the API accepts that certificate — forwarded by the TLS terminator in x-client-cert-fingerprint (or x-client-cert-pem) with x-client-cert-verify: SUCCESS — as the credential. A stolen bearer token cannot exist because there is none, and one certificate backs one active credential.
curl -sS -X POST https://spiryra.example.com/api/service-accounts/svc_…/tokens \
-H "Authorization: Bearer $SPIRYRA_TOKEN" -H "Content-Type: application/json" \
-d "{\"name\":\"api client\",\"scopes\":[\"certificates.read\"],\"clientCertificate\":\"$(cat client.pem)\"}"
The terminator must overwrite those headers; set SPIRYRA_AGENT_MTLS_PROXY_SECRET to require a shared proxy secret when it cannot.
Create an account and token
curl -sS -X POST https://spiryra.example.com/api/service-accounts \
-H "Authorization: Bearer $SPIRYRA_TOKEN" -H "Content-Type: application/json" \
-d '{"name":"inventory sync","description":"nightly report"}'
curl -sS -X POST https://spiryra.example.com/api/service-accounts/svc_…/tokens \
-H "Authorization: Bearer $SPIRYRA_TOKEN" -H "Content-Type: application/json" \
-d '{"name":"report","scopes":["certificates.read","servers.read"],"expiresInDays":90}'
| Route | Purpose |
|---|---|
| GET /api/service-accounts | Accounts and token metadata; never secrets. |
| POST /api/service-accounts | Create an account, optionally bound to an owner team with teamId. |
| PATCH /api/service-accounts/{id} | Change disabled and/or teamId (send an empty string to clear the team); disabling also revokes every live token. |
| POST /api/service-accounts/{id}/tokens | Issue a scoped token, optionally with an expiry. |
| DELETE /api/service-accounts/{id}/tokens/{token_id} | Revoke one token immediately. |
How a token is authorized
- Administrator-only paths and methods are refused outright.
- The route is mapped to the action it performs; an unclassified route is refused.
- That action must appear in the token's scopes. Scopes match exactly — certificates.read cannot issue, renew or revoke.
Team-scoped accounts
An account without a team is instance-wide: a certificates.read token sees every certificate, which suits inventory export and reporting. Bind the account to an owner team instead and the same token reaches only the certificates, servers, workflows, connectors and agents that team owns — the action scope still applies, but the object scope is the team. A certificate owned by another team answers 404, exactly as it does for a scoped person. The team must already exist under Roles & Permissions; a misspelt id is refused rather than quietly narrowing the account to nothing.
Manage tokens in the console
Settings → Service Accounts (administrator only) manages the same API: an account list with its team scope, live-token counts and a Disable action that revokes every live token, a token list with scope summary, authentication method, expiry, last use and state, and an issue dialog. The dialog takes a name and lifetime, optionally binds a client certificate instead of issuing a secret, and shows scopes as checkboxes grouped by resource, built from the same permission catalogue the Roles page uses. Actions whose handlers require an administrator session are shown disabled with the reason, because a token can never reach those surfaces. A bearer plaintext appears once after issue, with a copy button; a certificate credential shows its fingerprint and no secret.
Full detail, including the credential format and the test coverage, is in SERVICE_ACCOUNTS.md.
9.7The administrator boundary
A set of surfaces is administrator-only no matter what grants or scopes an account holds. The boundary is explicit, method-scoped and audited by a static check.
Administrator-only prefixes
/api/access/ /api/users /api/settings /api/notifications
/api/selftest /api/agent-upgrades/ /api/agentless/
/api/connectors/scep/ /api/connectors/est/ /api/connectors/adcs/
/api/demo/ /api/enterprise/deployment/ /api/migration/
/api/service-accounts
Plus exact method/path pairs such as POST /api/renewals/run, POST /api/monitor/scan and GET /api/orchestration/approvals, and suffix rules such as /api/runs/{id}/approval.
Deliberate exceptions
- GET /api/notifications — an owner reads their own delivery history.
- GET /api/agent-upgrades/rollouts — read-only rollout status.
- GET /api/enterprise/deployment/recovery — read-only recovery state.
- POST /api/agentless/certificates/execute — scoped clientless delivery.
- Migration reconcile routes — method-scoped so a sibling verb cannot inherit the exception.
Why it is trustworthy
- Exceptions are listed per HTTP method, so adding a method to the same path does not widen access.
- An unclassified API route is refused for non-administrators instead of falling through.
- scripts/audit-rust-authorization.py fails the release gate if a route is not classified. It currently reports 219 route-method pairs with 0 needing classification.
10.1SMTP email
Configure the relay
- Open Settings → Notifications (or the Notifications page) and select SMTP.
- Enter host, port, TLS mode, sender address and the public console URL used in links.
- Enter the username and password. The password is sealed and never returned.
- Send a test message to an address you control.
| Setting | Notes |
|---|---|
| Host and port | Use 587 with STARTTLS, or 465 with implicit TLS, unless your relay is internal. |
| Sender | Must be an address your relay accepts; it appears in From. |
| Public URL | Used for links in recovery and alert mail; must match SPIRYRA_PUBLIC_ORIGIN. |
| Credentials | Stored sealed; rotate after vault rotation. |
Console settings take precedence over environment variables, so the relay can be changed without a restart. Password recovery (3.3) uses the same saved relay.
GET /api/notifications/settings
PATCH /api/notifications/settings
POST /api/notifications/test
10.2Webhook channels
Channels deliver the same events as email: expiry warnings, deployment failures, revocation notices and escalation. Slack, Teams, PagerDuty v2 and a generic JSON channel are supported.
Add a channel
- Open Notifications → Channels → Add and choose the type.
- Paste the webhook URL or routing key. Secrets are sealed and never returned.
- Choose which events the channel receives.
- Send a test and confirm the message in the target system.
GET /api/notifications/channels
POST /api/notifications/channels
POST /api/notifications/channels/{id}/test
DELETE /api/notifications/channels/{id}
Transport rules
- HTTPS only; redirects are refused.
- Private and loopback targets are blocked unless explicitly enabled.
- Delivery retries are recorded in the same history as email, with the last error preserved.
10.3Escalation and delivery history
Escalation
When a renewal or deployment keeps failing, escalation ensures a human is told. Configure a failure threshold and a fallback recipient list; when the threshold is crossed, the owner and fallback recipients are notified.
- Automatic renewal opens at most one escalation per failed renewal streak.
- A deployment episode escalates once, not on every retry.
- Revocation notice from the CA is a one-time event per certificate (8.3).
Delivery history
GET /api/notifications
POST /api/notifications/{id}/retry
Each entry records the channel, event, attempt count, outcome and last error. A failed delivery can be retried without re-running the triggering operation.
11.1Health and readiness
| Endpoint | Question it answers | Typical output |
|---|---|---|
| /healthz | Is the process serving? | ok |
| /readyz | Can it do its job? | 200 with per-check status, or 503 when a check fails. |
| /api/meta | What is this deployment? | Profile, available connector kinds, demoAvailable, productionReady: false. |
| /metrics | What is it doing? | Prometheus text: request counters, queue states and agent liveness. Not proxied by the web container. |
Readiness checks
- data-volume — the data directory is writable and present.
- database — SQLite opens and passes quick_check.
- schema — the database version is not newer than this binary. An older database is upgraded in place; a newer one reports 503 with the found and supported versions, because an old API must not read a row shape it does not know.
- identity — the account schema and password policy load. A failed migration reports 503 instead of leaving sign-in broken behind a healthy process.
- connectors — at least one CA connector is configured and reachable; a fresh install is degraded here, which is expected.
A fresh install therefore answers 200 degraded until a CA is connected. A failure in any check returns 503 — that is the signal to stop traffic, not to restart blindly.
curl -fsS https://spiryra.example.com/healthz and curl -i https://spiryra.example.com/readyz. The startup log prints the same summary plus each check's detail.Metrics
GET /metrics answers in Prometheus text format from the API process. The web container does not proxy it, so it is reachable from the container network and from whatever scrapes the API directly. Set SPIRYRA_METRICS_TOKEN to require Authorization: Bearer <token>; the production profile requires it, and without a token the endpoint answers anyone who can reach the API port.
| Series | Type | Meaning |
|---|---|---|
| spiryra_http_requests_total | counter | Responses by method, normalised route and status. Ids collapse to :id, so the label set stays bounded. |
| spiryra_agents | gauge | Enrolled, unrevoked agents by online/offline. |
| spiryra_fleet_jobs | gauge | Fleet queue rows by state, excluding Windows PFX imports. |
| spiryra_windows_pfx_imports | gauge | Windows PFX imports by state, including failed ones. |
| spiryra_uptime_seconds, spiryra_info | gauge | Process age and build identity. |
11.2Backup and restore
Back up a running profile
./scripts/backup-profile.sh test
./scripts/backup-profile.sh production
The script uses SQLite's online backup API for every *.sqlite file, verifies each snapshot with PRAGMA integrity_check, copies the remaining data-volume files, and writes a checksummed manifest.json. Default destination:
backups/<profile>-<UTC timestamp>/
Restore
./scripts/restore-profile.sh test backups/test-20261029T010203Z --yes
- Verify the manifest and every checksum.
- Take a fresh pre-restore backup automatically.
- Stop the API, replace the database files, restore ownership.
- Start the API and wait for /healthz.
Rehearse the production path
./scripts/verify-production-deploy.sh --check
./scripts/verify-production-deploy.sh --backup-only
./scripts/verify-production-deploy.sh --restore-drill --yes
The drill takes a backup, verifies checksums, restores the same archive, checks health and compares certificate and account counts. It causes a short outage and keeps the pre-restore backup.
11.3Vault key rotation
Changing SPIRYRA_VAULT_KEY on its own makes every sealed value unreadable. Rotate in three steps instead.
- Introduce the new key alongside the old. Set the new value in SPIRYRA_VAULT_KEY and put the old value in SPIRYRA_VAULT_KEY_PREVIOUS (comma-separate several), then redeploy. Reads and new writes keep working.
- Re-wrap. As an administrator open Settings → General → Vault key rotation and confirm Re-wrap sealed values. One transaction decrypts and re-encrypts retained leaf keys, connector credentials, registered-server SSH keys, WinRM passwords, notification secrets, the SMTP password and the OIDC client secret. If any value cannot be opened, nothing changes and the response names the affected table.
- Retire the old key. Remove SPIRYRA_VAULT_KEY_PREVIOUS, redeploy, and back up the new key separately.
11.4Background workers and leases
Renewal scheduling, revocation checks, TLS monitoring, deployment dispatch and orchestration all run inside the API process. Work is claimed through database leases so two replicas do not do the same job twice.
- Orchestration runs hold a lease while a replica drives them; a second replica cannot pick up the same run.
- Two API replicas were exercised and split background work into distinct leases.
- On startup a replica resumes queued and running runs, reattaches host steps to their existing agent jobs and fails in-flight control-plane steps closed.
- An offline agent leaves its deployment queued — there is no automatic replay, so a queued job is a visible state, not a silent retry loop.
11.5Audit log and events
Every security-relevant action is written to an append-only audit log with actor, action, severity, summary and structured detail. The console reads it as an event stream.
GET /api/events
GET /api/access/audit
Representative actions
| Action | Severity | When |
|---|---|---|
| certificate.issued | info | A CA returned a validated certificate. |
| certificate.revoked | warning | The CA confirmed a revocation. |
| certificate.ca_revoked | warning | A CA-side revocation was detected by the worker. |
| service_token.created | warning | A service account token was issued. |
| service_token.revoked | warning | A token was revoked. |
| batch.rolled_back | warning | A batch rolled back, with the manual actions listed. |
| notification.settings_updated | info | The delivery configuration changed. |
What audit never stores
- Private keys, tokens, passwords or sealed values.
- ACME account keys and DNS API tokens.
- Full certificate bodies beyond what the record needs.
11.6Logging and diagnostics
Container logs
docker compose -p spiryra-production logs -f api
docker compose -p spiryra-production logs -f web
On startup the API prints the readiness summary and each check's detail. Connector and deployment errors appear with the CA or host message included.
What to collect before asking for help
- The output of /readyz and /api/meta.
- The failing request's status code and the JSON body — errors carry a message, and authorization failures carry a code.
- The relevant container log lines, with secrets redacted.
- For deployment failures: the deployment detail, including reload result and presented fingerprint.
- For connector failures: the connector Test output, not just the issue error.
12.1Security model
| Property | How it is enforced |
|---|---|
| No internal CA in production | The release build has no in-process CA; production refuses a local-ca kind or demo seed. The test CA exists only in the test profile. |
| No private keys on targets | Deployment writes leaf, chain and full-chain bundle only. Retained keys require an administrator-gated download. |
| Secrets at rest | AES-256-GCM with SPIRYRA_VAULT_KEY; connector secrets are bound to the connector id; the API returns field names, never values. |
| Session integrity | SHA-256 digests in clm_sessions; CSRF token bound to the session; same-origin check on browser writes. |
| Authorization depth | Role check, resource grants, route classification gate, and handler-level ownership checks — each independently enforced. |
| Host-side limits | Agent install roots, approved commands, approved services and TLS probe allowlists. The control plane cannot widen them. |
| Outbound TLS | Verification is mandatory; redirects refused; embedded URL credentials rejected; no insecure-skip switch. |
| Auditability | Append-only events for issuance, revocation, tokens, approvals, rollbacks and settings changes. |
Trust boundaries
- The Docker host is trusted: it holds the env file and the vault key.
- The CA is authoritative for policy and validity; Spiryra validates what comes back but does not overrule it.
- The target host is semi-trusted: it enforces its own allowlists and reports what it did.
- The browser is untrusted: every write is authorized server-side, and tokens never reach JavaScript beyond the CSRF value.
12.2Hardening checklist
- Use the production profile with an exact HTTPS public origin and secure cookies.
- Keep port 8080 on loopback; only the TLS proxy is reachable.
- Lock the env file to mode 0600 and store the vault key in your secret manager separately from backups.
- Replace the bootstrap administrator with named accounts; keep one local administrator for IdP outages.
- Use HTTPS for WinRM (5986) and keep SPIRYRA_AGENTLESS_WINRM_ALLOW_HTTP unset.
- Set agent allowlists explicitly and leave them empty where the capability is not needed.
- Scope tokens narrowly: service accounts get read scopes unless they must write, plus an expiry.
- Turn on notifications and escalation so expiry and failures reach a person.
- Rehearse restore and vault rotation before you need them.
- Run the release gate (14.3) before each production change window.
12.3Known limits
| Area | Limit |
|---|---|
| Agent transport | Device attestation is not implemented. Credentials are scoped per agent (see 6.2) and can be bound to a client certificate, which makes a stolen token useless without that certificate; an unbound credential is still a bearer secret, so a stolen token acts within its own host's capabilities until it is rotated or revoked. |
| Agent packaging | DEB and RPM packages are built by scripts/package-agent.sh, carry the agent, a separate supervisor and systemd units, and are installed and removed in throwaway Debian and EL9 containers by the gate (agent-packages). They are not signed by a distribution or with Authenticode, so integrity rests on SHA256SUMS and its detached signature when a release key is configured. Windows ships an executable archive with the scheduled-task installer, not an MSI. Upgrades still activate a verified executable rather than running the package manager. |
| Enrollment | Self-enrollment is rate limited per client address (30/hour by default) and sign-in at 10 per 5 minutes. The windows live in the API database, so replicas that share it share one budget. The client address is the socket peer unless the peer is listed in SPIRYRA_TRUSTED_PROXIES, which is what makes a proxy's X-Real-IP believable; keep a proxy limit in front of enrollment as well. |
| Fleet listing | Agents and the recent-job list beside them both page (limit/offset and jobLimit/jobOffset; agents default 500, max 1000; jobs default 25, max 200), each with its own nextOffset cursor. The console shows both totals, so older jobs are reachable. |
| SCEP | One common name per request, no revocation. |
| EST | No revocation operation in the protocol. |
| AD CS Web Enrollment | Issuance only — no revocation. |
| EJBCA approvals | An approval-gated enrollment is recorded instead of lost: Spiryra keeps the EJBCA approval request id (and the sealed key when it generated one) and polls POST /certificate/{id}/finalize until an RA approves it, then records the certificate through the normal inventory path. Polling stops after SPIRYRA_EJBCA_APPROVAL_MAX_POLLS failed attempts and the row is marked expired. The collector, the real finalize call, recorded errors and expiry ran against a live EJBCA 9.6 CE CA; an approval-gated certificate profile is an EJBCA admin-UI setting, so the approve-then-issue leg is covered by tests rather than a live RA approval. |
| Alerts | An alert row now keeps its own lifetime: first and last sighting, the last observed expiry data, and its status. A scan refreshes the observation, reopens an alert whose condition returns, and reopens an acknowledged alert that escalates from warning to critical; resolving an alert whose certificate has left the warning window keeps it readable under the status filters. The list still merges the live expiry projection with those rows, so an alert appears as soon as its certificate enters the window, without waiting for a scan. |
| Canonical records | The list stays a metadata projection. Opening one migrated row reads its stored cert_pem (the public certificate) and derives SANs, key detail, validity start and PEM from it; the certificate download serves it. Event history is not migrated, there is no intermediate chain in the canonical schema, and a database that predates cert_pem still opens metadata-only. No private key column is ever read. |
| Service accounts | An account can be bound to one owner team, so its token reaches only that team's certificates, servers, workflows, connectors and agents; unbound accounts stay instance-wide. A token can also be a certificate credential with no secret, resolved from the client certificate the TLS terminator reports — the same header trust model as agent mTLS, so the terminator must overwrite those headers (or a shared proxy secret must be configured). One certificate backs one active credential. |
| OCSP | The worker's request path is exercised against a real OpenSSL responder (good and revoked certificates) in cargo test, so a broken request, a nonce the responder rejects or a status the parser mishandles fails the build. The responder is generated by the test; it is not your issuer's, and the scheduled refresh still needs an operator run against a production CA. |
| WinRM delivery | File install, restart and presentation are verified as scripts, and a staged PFX import is verified end to end from the API against a Windows container (scripts/verify-winrm-pfx-e2e.py). The four orchestration steps are also proven against a live Windows Server 2025 host (scripts/verify-winrm-live-host.py): store import, HTTP.sys rebind, live handshake, a real W3SVC restart and a probe of the served certificate. Private-key ACL binding as a service identity remains agent-side: scripts/verify-windows-pfx-identity-e2e.py runs the agent's own import on a live Windows Server 2025 host and then signs with the bound identity, so the binding is proven to grant real use of the key, not just the right ACE. |
| Windows PFX | Effective access is proven, not just the ACE. On a live Windows Server 2025 host the agent's own import binds the key to NT AUTHORITY\NETWORK SERVICE; a probe running as that identity signs with the key (a real 2048-bit signature), NT AUTHORITY\LOCAL SERVICE is refused, and the control leg binds that second identity and has it sign, so the refusal is the missing ACE rather than an identity that cannot use CNG keys. The negative leg reports the refusal as Invalid provider type specified rather than a plain access-denied message, so the assertion is that no signature is produced. Still unproven: a domain service account, and IIS serving with a key ACL'd to a non-administrator pool identity. |
13.1API authentication and CSRF
Credential types
| Credential | Header | Use |
|---|---|---|
| Console session | Cookie: spiryra_session | Browser console; writes also need the CSRF header. |
| Migration admin token | Authorization: Bearer … | Bootstrap, scripts, recovery. Administrator identity. |
| Agent token | Authorization: Bearer … | This host's job feed and results only, limited to the capabilities its credential holds (6.2). |
| Service account token | Authorization: Bearer csa.<id>.<secret> | Scoped automation (9.6). |
Sign in
POST /api/auth/login
{ "username": "admin", "password": "…" }
200 OK
Set-Cookie: spiryra_session=…; HttpOnly; SameSite=Strict
Set-Cookie: spiryra_csrf=…; SameSite=Strict
{ "user": { "username": "admin", "role": "admin" }, "csrfToken": "…" }
Write requests from a browser
POST /api/certificates/issue
Cookie: spiryra_session=…
X-CSRF-Token: <value from the csrfToken field or the spiryra_csrf cookie>
- A missing or wrong CSRF token returns 403.
- The request Origin must match the request Host; a mismatch returns 403.
- Bearer credentials — migration tokens, agents and service accounts — are not subject to the browser CSRF rule.
POST /api/auth/logout
GET /api/auth/me
GET /api/auth/providers
13.2Endpoint reference
Representative routes grouped by capability. Every console screen uses these; the authorization gate classifies each route-method pair before a handler runs.
Certificates
GET /api/certificates
GET /api/certificates/{id}
POST /api/certificates/issue
POST /api/certificates/csr/sign
POST /api/certificates/import
POST /api/certificates/{id}/renew
POST /api/certificates/{id}/revoke
GET /api/certificates/{id}/download[?part=chain|key]
GET /api/certificates/{id}/installations
GET /api/certificates/{id}/automation
Connectors and CA policy
GET /api/connectors
GET /api/connectors/specs
POST /api/connectors
PATCH /api/connectors/{id}
POST /api/connectors/{id}/test
GET /api/connectors/{id}/templates
POST /api/connectors/{id}/templates/sync
POST /api/connectors/est/inspect POST /api/connectors/est/enroll
POST /api/connectors/scep/inspect POST /api/connectors/scep/pki-operation
GET /api/ca/templates
Deployment and fleet
GET /api/installations POST /api/installations
PATCH /api/installations/{id}
POST /api/installations/{id}/deploy
POST /api/installations/{id}/rollback
GET /api/agents POST /api/agents
POST /api/agents/bulk GET /api/agents/fleet
DELETE /api/agents/{id}
GET /api/agent/jobs POST /api/agent/jobs/{id}/result
POST /api/agent-bootstrap/enroll
POST /api/agent/token/prepare POST /api/agent/token/commit
GET /api/agent-upgrades/rollouts
POST /api/agent-upgrades/rollouts/{id}/dispatch
POST /api/agent-upgrades/rollouts/{id}/approve
POST /api/agent-upgrades/rollouts/{id}/cancel
GET /api/agent-upgrades/rollouts/{id}/outcomes
Orchestration
GET /api/orchestration/meta
GET /api/orchestration/approvals
GET /api/workflows POST /api/workflows
GET /api/runs POST /api/runs
GET /api/runs/{id}
POST /api/runs/{id}/approval
Discovery, monitoring and alerts
GET /api/findings PATCH /api/findings/{id}/state
POST /api/findings/{id}/import
GET /api/monitor/tls POST /api/monitor/tls
GET /api/monitor/tls/{id}/history POST /api/monitor/scan
GET /api/monitor/tls/alerts POST /api/monitor/tls/alerts/{id}/acknowledge
GET /api/alerts
POST /api/alerts/{id}/ack POST /api/alerts/{id}/resolve
Identity and access
GET /api/users POST /api/users
PATCH /api/users/{id} DELETE /api/users/{id}
GET /api/access/summary GET /api/access/catalog
GET /api/access/directory GET /api/access/preview
POST /api/access/teams POST /api/access/owners
POST /api/access/roles POST /api/access/group-roles
GET /api/access/ca-policy POST /api/access/ca-policy
GET /api/access/audit
GET /api/service-accounts POST /api/service-accounts
PATCH /api/service-accounts/{id}
POST /api/service-accounts/{id}/tokens
DELETE /api/service-accounts/{id}/tokens/{token_id}
Operations and settings
GET /api/meta GET /api/dashboard
GET /api/settings PATCH /api/settings
GET /api/selftest
GET /api/events
GET /api/notifications GET /api/notifications/settings
PATCH /api/notifications/settings POST /api/notifications/test
GET /api/notifications/channels POST /api/notifications/channels
POST /api/notifications/channels/{id}/test
POST /api/notifications/{id}/retry
GET /api/renewals POST /api/renewals/run
POST /api/renewals/{certId}/retry
GET /api/orders POST /api/orders/{id}/continue
GET /api/enterprise/deployment/backups
POST /api/enterprise/deployment/backups
GET /api/enterprise/deployment/recovery
13.3Errors and status codes
Errors are JSON. Most carry a message; authorization failures also carry a code.
{ "code": "RBAC_DENIED", "message": "The role or owner team does not grant access to this operation" }
{ "message": "Unknown scope: certificates.superuser" }
| Status | Meaning | Typical cause |
|---|---|---|
| 400 | Invalid request | Bad field, unknown scope, invalid expiry, malformed CSR. |
| 401 | Not authenticated | Missing or expired session, revoked or expired token, wrong secret. |
| 403 | Not authorized | Role, grant, scope or administrator boundary; missing/incorrect CSRF; origin mismatch. |
| 404 | Not found | Unknown id — also returned for resources outside your scope. |
| 409 | Conflict | Duplicate name, alert/rollout already actioned, EJBCA approval still pending, disabled account. |
| 502 | Upstream failure | The CA or target rejected the request; the message preserves the CA's answer. |
| 503 | Not ready | Database, identity store or required configuration missing — check /readyz. |
14.1What is verified
Everything below was executed against this source tree or a recorded lab environment. Anything not listed here is covered in 14.2 — treat it as unproven.
| Area | Evidence |
|---|---|
| Rust API and console | cargo test: 283 passed, 0 failed, including authorization, session, connector, deployment, orchestration and service-account suites. |
| Go agent | go vet ./..., go test ./... and go test -race ./... pass. |
| Route classification | scripts/audit-rust-authorization.py: 219 route-method pairs, 0 needing explicit classification. |
| API parity | scripts/audit-api-parity.py: 112 frontend paths, 112 registered routes, 0 missing (static check, not behavioural parity). |
| Test profile stack | Both images built, API healthcheck passed, console served; sign-in, settings, inventory, orders, events, issuance with retained key, idempotent continuation, 7/7 self-test and revocation executed through nginx. |
| Clientless SSH | Delivery, verification and rollback executed end to end on the lab. |
| Clientless WinRM | Install, same-key upgrade and rollback verified in a Windows Server 2025 container; reload and TLS presentation scripts verified with five checks and zero failures. |
| AD CS issuance | WinRM issuance verified end to end against a Windows Server 2025 CA; Web Enrollment issuance verified against a domain CA with the role installed. |
| Windows PFX | Import into LocalMachine/My with hasPrivateKey=true and a read ACE for a non-admin service identity, in a Server Core container; then use of that key by the bound identity, and refusal for an identity without the ACE, on a live Windows Server 2025 host. |
| IIS changeout | Rebind a wildcard and an SNI HTTPS binding in a Windows Server 2025 IIS container, read HTTP.sys back, and fail an ambiguous binding without disturbing the real one. A site with no HTTPS binding was given one and served the import on a live handshake; a stopped site's new binding and HTTP.sys entry were removed when the job could not confirm the changeout. A leaf issued through an intermediate CA under the lab root was imported into an application site with its own app pool, served 200 with the application body, built a three-level chain, and survived an app-pool recycle. A queued changeout then recycled that pool itself — the worker pid changed, the old process drained and exited, and the site kept serving the same certificate. |
| WinRM PFX import | Request validation and script substitution are tested; a live import needs a reachable WinRM host with staged PFX material. |
| EJBCA connector | Lab lifecycle: create, test, issue, renew and revoke through the REST API. |
| Vault PKI connector | Vault 1.18.5: create, update, test, ECDSA P-256 issue, OpenSSL chain verification, renew with a fresh key, revoke and CRL check. |
| ACME v2 | Local stub: HTTP-01 issue and revoke, wildcard DNS-01 with TXT creation and cleanup, rejected TXT, rejected cleanup, and journal recovery after an API kill. |
| Revocation worker | Checked a real Google Trust Services CRL and a live OpenSSL OCSP responder reporting both good and revoked. |
| Replicas | Two API replicas split background work into distinct leases. |
| Backup and restore | Destructive drill restored certificates, accounts, audit events and connector credentials. |
| Vault rotation | Live rotation rewrapped six sealed values with zero skips; connectors still opened them after the previous key was removed. |
| Service accounts | Scoped token reaches a granted read, is refused a classified write and four administrator surfaces, digest-only storage, closure on revoke/expiry/disable. |
14.2What still needs your environment
| Open item | Why it is open | What closes it |
|---|---|---|
| Live EST and SCEP | No production-like EST or SCEP authority was available. | Issue, renew and revoke against your own EST/SCEP endpoints. |
| Public ACME | Only local stubs and preflight checks were run. | A real Let's Encrypt or commercial run with the port-80 route in place. |
| WinRM reload plus presentation | One Docker engine cannot run a Linux control plane beside Windows containers. | One control-plane-driven deployment against a reachable Windows target. |
| PFX effective access | Server Core refuses non-admin logons (0xC0000142). | Confirm the service identity can open the key on a real or domain-joined host. |
| WinRM PFX import | No reachable WinRM host with staged PFX material in the lab. | Run the staged import and IIS changeout against a real WinRM target. |
| AD CS with your policy | The lab proves the transports, not your templates, issuance policy or RA rules. | Issue against your template and confirm the returned chain. |
| Live OCSP with your issuer | The responder path is proven against a real OpenSSL responder in the suite, but the URL comes from the certificate, so only your CA's own responder proves your estate. | Run the refresh against your issuer and confirm a revoked result reconciles that certificate. |
| Continuous presentation monitoring at scale | Verified per endpoint, not across a large estate. | Run the monitor against your real endpoints for a full renewal cycle. |
| Independent security review | Not performed. | A signed review attestation accepted by the production gate. |
14.3Release gate
# staging gate: tests, builds, source manifest, optional signature
bash scripts/release-gate.sh
# production gate: also requires a separately signed independent review
bash scripts/release-gate.sh --production
python3 scripts/verify-release.py
npm run typecheck
npm run verify:settings-route
What the stages check
| Stage | Checks |
|---|---|
| Tests | cargo test --locked in the Rust backend, plus the repository's verification scripts. |
| Windows evidence | scripts/windows-pfx-gate.sh runs the WinRM PFX end-to-end harness: a Windows Server container, a staged import over WS-Management, the HTTP.sys rebind and a live TLS handshake. A host that cannot run Windows containers records a skip with the reason; --production and the scheduled Windows job (SPIRYRA_REQUIRE_WINDOWS_E2E=1) refuse to pass on a skip, because the control plane runs on Linux but deploys certificates to Windows targets. |
| Builds | API binary and console build; no build-only secrets in the manifest. |
| Manifest | Source digest recorded so a review can be bound to an exact tree. |
| Production review | Refuses a review for a different source digest, an identical signing key, open high/critical findings or missing external-validation evidence. |
verify:settings-route is a Playwright check for the canonical Identity Providers route and the legacy redirect. It needs a running test profile and either local administrator sign-in or the deployment migration admin token.
15.1Installation and sign-in problems
| Symptom | Cause | Fix |
|---|---|---|
| production check fails on the origin | Loopback, IP address, localhost or a path in SPIRYRA_PUBLIC_ORIGIN. | Use the exact browser-facing HTTPS DNS origin, no trailing path. |
| Preflight refuses the env file | Permissions are not 0600, or tokens are equal or shorter than 32 characters. | chmod 600 the file; regenerate distinct secrets. |
| /readyz returns 503 | Data volume, database or identity check failed. | Read the failing check in the response and the startup log before restarting. |
| readyz says degraded | No CA connector configured yet. | Expected on a fresh install; add and test a connector. |
| Sign-in says invalid credentials on a fresh test stack | Bootstrap hash not applied, or a reused database with a different password. | Run reset-test-admin-password.sh (test only) or redeploy with the intended password. |
| Every write returns 403 | Missing CSRF header, or the proxy rewrote Host so the origin check fails. | Send X-CSRF-Token; preserve Host in the proxy. |
| Console loads but API calls 502 | API container unhealthy or not started. | docker compose -p <project> ps and inspect the api logs. |
| Browser warns about an insecure cookie | Test profile over HTTP. | Expected in test; never enable the lab switch in production. |
15.2Connector problems
| Symptom | Likely cause | Fix |
|---|---|---|
| Test fails with a TLS error | Private root not in the connector trust bundle. | Add the root PEM to deploy/ca/ and reference it; do not look for a skip-verify switch (there is none). |
| EJBCA test passes but issue fails | Missing create-certificate or end-entity-profile rule. | Grant the rules listed in 4.2. |
| Revoke returns 502, certificate still active | Missing revoke rule at the CA. | Grant revoke on the CA and end-entity profile, then retry. |
| Issue rejected: “requested SAN is missing” | The CA profile strips or forbids the SAN. | Change the profile/role, or request only permitted names. |
| Vault issue rejected on TTL | Role max_ttl shorter than requested. | Request within the role's limit; the CA is authoritative. |
| ACME wildcard refused | Connector has no DNS zone, so only HTTP-01 is available. | Configure dnsZoneId, dnsZoneName and dnsApiToken. |
| ACME HTTP-01 times out | Port 80 not forwarded to the web container. | Add the /.well-known/acme-challenge/ route in the proxy. |
| EST/SCEP unavailable | Host not in the allowlist or the profile was not redeployed. | Set SPIRYRA_EST_ALLOWED_HOSTS / SPIRYRA_SCEP_ALLOWED_HOSTS and redeploy. |
| AD CS Web Enrollment test fails | Account cannot enrol against the template, or the site certificate is untrusted. | Grant enroll permission; set caPem to the site's trust anchor. |
15.3Deployment problems
| Symptom | Cause | Fix |
|---|---|---|
| Delivery stays queued | Agent offline. Jobs do not replay automatically. | Bring the agent online; the queued job is dispatched on its next poll. |
| copy_file denied | Path outside SPIRYRA_INSTALL_ROOTS. | Widen the roots on the host, or choose a path inside them. |
| reload denied | Service not in SPIRYRA_APPROVED_SERVICES. | Add the service on the host; it is not a control-plane setting. |
| Probe denied | Endpoint not in SPIRYRA_TLS_PROBE_ALLOWLIST (empty denies all). | Add the exact host:port on the host. |
| File digest mismatch | Write partially failed, or something rewrote the file. | Retry; inspect the destination and any local automation on the host. |
| Fingerprint mismatch after reload | Service did not actually reload, or a different certificate is served. | Check the service and the endpoint's SNI; an automatic rollback is queued when the key did not change. |
| WinRM connection refused | WinRM disabled, or 5985 used in a profile that forbids HTTP. | Enable WinRM; use 5986 with a trusted listener in production. |
| PFX import fails on ACL | Service identity unresolvable, or the ACE could not be written. | Fix SPIRYRA_WINDOWS_PFX_SERVICE_ACCOUNT; the job fails closed by design. |
| Rollback marks restored but the service serves the old cert | Reload was not approved or did not run. | Re-issue the reload, then verify the presented fingerprint. |
15.4Orchestration problems
| Symptom | Cause | Fix |
|---|---|---|
| A WinRM server is reported as not runnable | The host is not in SPIRYRA_AGENTLESS_WINRM_HOSTS, or the password, username or host is missing. | Add the host to the API-host list and store the sealed password on the server. |
| WinRM step is refused with an allowlist message | The path, command, service or endpoint is not in the SPIRYRA_WINRM_* list. | Change the step or widen the API-host list — commands match exactly and paths must sit inside a listed root. |
| SSH step fails before it connects | SPIRYRA_AGENTLESS_SSH_KNOWN_HOSTS is unset, or the server's host key is not pinned. | Pin the key on the API host; a non-default port needs the [host]:port form. |
| SSH step is refused with an allowlist message | The path, command, service or endpoint is not in the SPIRYRA_SSH_* list. | Change the step or widen the API-host list — the match is literal. |
| Discovery step is refused with 409 | Discovery needs the agent's inventory walk. | Assign an agent to that server, or drop the discovery step. |
| Run waits indefinitely | An approval gate is pending, or a host step's agent is offline. | Check Approvals; gates time out rather than proceeding. |
| Command step fails before execution | The command does not exactly match SPIRYRA_APPROVED_COMMANDS. | Change the step or the host allowlist — the match is literal. |
| Batch did nothing | A target was invalid, so atomic creation refused the whole batch. | Fix the target; the refusal is intentional. |
| Batch halted mid-way | A target failed and siblings were stopped before their next step. | Read the batch detail: restored, reloaded and manual lists. |
| Recovery ledger still shows manual steps | Commands and API calls have no automatic inverse. | Review each entry and close it by hand; it will not clear itself. |
| Run restarted from the beginning | Database lease lost, or a second replica took over. | Check logs for lease recovery; succeeded steps should be preserved. |
15.5Diagnostic commands
# Stack state and logs
./scripts/deploy-profile.sh production status
docker compose -p spiryra-production ps
docker compose -p spiryra-production logs --tail=200 api
# Liveness and readiness
curl -fsS https://spiryra.example.com/healthz
curl -sS -i https://spiryra.example.com/readyz
curl -sS https://spiryra.example.com/api/meta | head -c 400
# Backups
./scripts/backup-profile.sh production
ls -1 backups/ | tail -3
# Recovery rehearsal (short outage)
./scripts/verify-production-deploy.sh --restore-drill --yes
# Authorization and parity audits
python3 scripts/audit-rust-authorization.py
python3 scripts/audit-api-parity.py
16Glossary
ACME — Automatic Certificate Management Environment (RFC 8555). Spiryra supports HTTP-01 and DNS-01, including wildcards, with optional External Account Binding.
Agent — the native Go executable installed on a managed server. It polls the control plane, writes certificate files, reloads approved services and reports the certificate a socket presents.
Agentless (clientless) — delivery over SSH or WinRM with no Spiryra software on the target.
Allowlist (agent) — host-side lists that decide what a deployment may write, run, reload or probe. The control plane cannot widen them.
Approval gate — a workflow step that stops a run until an administrator approves or denies it.
Batch — one workflow run across several targets, created atomically.
CA policy — the record of which templates and key/validity constraints a deployment may use.
Connector — the configured path to an external authority (EJBCA, Vault PKI, ACME, AD CS, EST, SCEP), with sealed credentials.
CSRF token — a per-session value required on browser-originated writes (X-CSRF-Token).
Deployment target (installation) — a certificate bound to a server and destination path, with optional service reload and TLS endpoint.
EAB — External Account Binding, an ACME mechanism binding an account to a provider-issued key identifier and HMAC key.
EST — Enrolment over Secure Transport (RFC 7030).
Fingerprint — the SHA-256 digest of a certificate (or file), used to prove what was written and what is served.
Grant — a per-resource permission binding an account to a certificate or connector with read or manage rights.
Installation roots — the directories an agent may write into (SPIRYRA_INSTALL_ROOTS).
OCSP / CRL — the two revocation status mechanisms the worker polls for active certificates.
PFX / PKCS#12 — the Windows certificate container the agent can import into the machine store.
Readiness — /readyz: whether the deployment can actually do its job, not merely whether the process is alive.
Recovery ledger — the record of batch actions that have no automatic inverse, kept for operator follow-up.
Retained key — a private key stored sealed in the vault, enabling key-reuse renewal and administrator-gated download. Never deployed.
Run — one workflow execution against one target, with per-step state and evidence.
SCEP — Simple Certificate Enrolment Protocol (RFC 8894). One common name per request, no revocation.
Scope (account) — all or assigned; combined with grants to decide which resources an account may act on.
Scope (service account) — an action from the permission catalogue granted to a machine token; matched exactly.
Sealed value — a secret encrypted with AES-256-GCM under SPIRYRA_VAULT_KEY.
Vault key — the wrapping key for every sealed value. Back it up separately from data backups.
Workflow — the ordered definition of steps, targets and conditions that a run executes.
17Document index
Every claim in this guide is backed by a repository document. Start with the first two rows for status and deployment.
| Document | Contents |
|---|---|
| NATIVE_RELEASE_STATUS.md | Verification passes, gaps and release classification. |
| DEPLOY_FRESH.md | Profiles, manual env, TLS proxy, backup, rotation, removal. |
| RELEASE_GATE.md | Signed staging and production gate, manifest and review contract. |
| SERVICE_ACCOUNTS.md | Machine credentials, scopes, admin API and tests. |
| EXTERNAL_CA_ONLY.md | Why Spiryra is an interface to your CAs, not a CA. |
| RUST_EJBCA_CONNECTOR_MIGRATION.md | EJBCA REST lifecycle, roles and validation. |
| RUST_VAULT_PKI_CONNECTOR.md | Vault PKI connector and lab proof. |
| RUST_ACME_CONNECTOR.md | ACME v2, HTTP-01, EAB, DNS-01 and rechecks. |
| CLIENTLESS_ADCS_TRANSPORTS.md | AD CS over WinRM and Web Enrollment, with evidence. |
| ACME_DNS01_EST_WORK_STATUS.md | EST work status and DNS-01 detail. |
| SCEP_AGENTLESS_IMPLEMENTATION_STATUS.md | SCEP implementation and limits. |
| AGENTLESS_DEPLOYMENT_E2E.md | SSH and WinRM delivery, verification and rollback evidence. |
| AGENTLESS_SSH_WINRM_STATUS.md | Current agentless status and remaining checks. |
| WINDOWS_CONTAINER_TEST_EVIDENCE.md | Windows Server 2025 container evidence for WinRM and PFX. |
| WINDOWS_PFX_IMPLEMENTATION.md | PFX import and service-account ACL binding. |
| RUST_AGENT_ENROLLMENT_MODULE.md | Agent registry, enrollment, rotation and limits. |
| AGENT_UPGRADE_DISPATCH_STATUS.md | Signed release catalog and supervisor. |
| RUST_ORCHESTRATION_MODULE.md | Canvas, runs, parallel groups, approvals and batches. |
| RUST_RENEWAL_DEPLOYMENT.md | Renewal plus deployment evidence and timeouts. |
| RBAC_SECURITY_WORK_COMPLETED.md | Roles, teams, ownership, scoped reads, admin boundary. |
| RBAC_ADMIN_BOUNDARY_REVIEW.md | Why each administrator surface and exception exists. |
| PASSWORD_MANAGEMENT_STATUS.md | Password policy, lockout, history and recovery. |
| SMTP_ESCALATION_E2E_PROOF.md | Notification delivery and escalation evidence. |
| CONNECTOR_E2E_PROOF.md | Connector end-to-end lab proof. |
| RELEASE_VERIFICATION_2026-10-30.md | What was executed for the current closeout, and what was not. |
| IMPLEMENTATION_PROGRESS.md | Progress narrative across the native migration. |