Skip to content

ADR-016: One explicit browser TLS certificate per PR preview

Context

On 2026-08-13 every preview browser Ingress independently carried the letsencrypt-prod ingress-shim annotation and a distinct TLS Secret. The seven public preview hosts therefore produced up to seven single-SAN certificates per preview. Let’s Encrypt returned HTTP 429 after 50 certificates for syrf.org.uk had been issued in its preceding 168-hour window. PR #2809 then served the ingress controller's fake certificate for API, Project Management, and User Guide while Argo applications still reported Synced and Healthy.

The same failure class recurred on 2026-08-30 in PR #2992. Five public single-host Certificates became Ready, while Docs failed separately and the Web TLS Secret was absent after an ACME finalization race. Nginx served its built-in fake certificate for Web even though the workflow and Argo health had already reported success. This proves that application health and independent per-service issuance are not a browser-readiness contract.

The production issuer is verified HTTP-01 only. DNS-01 and wildcard issuance are not available infrastructure capabilities.

Decision

Each preview will own one explicit cert-manager.io/v1 Certificate in the preview-infrastructure chart. It has exactly these seven DNS names:

  • pr-{N}.syrf.org.uk
  • api.pr-{N}.syrf.org.uk
  • project-management.pr-{N}.syrf.org.uk
  • identity.pr-{N}.syrf.org.uk
  • quartz.pr-{N}.syrf.org.uk
  • docs.pr-{N}.syrf.org.uk
  • help.pr-{N}.syrf.org.uk

The Certificate owns deterministic Secret pr-{N}-tls. The preview infrastructure chart enables it by default with the established letsencrypt-prod ClusterIssuer. That issuer is Ready and uses the nginx HTTP-01 solver; the existing preview DNS records already route all seven names to that ingress class. This decision does not change production host Certificates, production DNS, or issuer configuration.

Every service Application already receives the authoritative numeric syrf.prNumber. The shared and bespoke Ingress templates use that preview identity to replace legacy per-service secretName inputs with pr-{N}-tls and remove cert-manager/ACME issuance annotations. This makes Certificate creation and consumption one chart-revision contract rather than a separately timed GitOps opt-in, and prevents ingress-shim from creating competing per-service Certificates. When no valid preview identity exists, staging and production rendering is unchanged.

Preview success is an end-to-end contract, not an Argo health inference. The PostSync notifier must verify the Certificate's Ready condition, its expected Secret name and exact SAN set, absence of competing browser Certificates, the exact seven-host Ingress-to-Secret mapping, and absence of issuer annotations. It then validates the Secret type/key shape, the leaf's validity and exact SAN set, trusted HTTPS for every public host, and that every endpoint serves the same leaf stored in the shared Secret. No Secret or private-key data is logged. Only after all checks pass may it update PR metadata and GitHub deployment and commit statuses to Ready. Build completion remains awaiting TLS verification; a failed or incomplete verifier reports failure or remains pending, never Ready. Disabling TLS while the readiness notifier is enabled is a Helm render error, so there is no fake-certificate success path.

This is intentionally not feature-flagged: it is a preview-only deployment safety contract with a GitOps revert boundary, not user-facing behaviour.

Consequences

  • Normal preview steady state changes from up to seven browser certificate issuances to one. Chart/workflow iteration uses renders and unit tests. A deliberately authorized trusted-chain proof may use one real preview issuance through the established issuer; it must not create extra preview Certificates or touch production hosts.
  • Wildcards are rejected: HTTP-01 cannot issue them, no DNS-01 capability is verified, and *.syrf.org.uk does not cover multi-label preview hosts.
  • Preview, staging, and production never share a browser TLS Secret. That would violate namespace/key isolation and widen renewal and credential blast radius.
  • Ordinary rebuilds reuse the deterministic Secret and Certificate. cert-manager owns renewal in place. A real namespace teardown/recreation necessarily reissues once, and readiness remains closed until the replacement Secret and all served leaves agree. Concurrent previews use disjoint PR-scoped names.
  • Rollback is a reviewed Git revert of the SyRF chart revision. It must not delete Certificates, Secrets, or previews manually and must preserve the no-premature-Ready gate.

Follow-up

Stale previews are a separate governance concern. Existing scheduled sweep paths are dry-run and their imperative mutation mode must not be enabled as a shortcut. A subsequent reviewed policy will use GitOps preview-directory reconciliation and approval-gated deletion, with an explicit lease/retention decision for old open labelled PRs.