Bulk PDF Agent ARRNC Hosting Correction — Design

Decision: ADR-015. Executable plan: 2026-08-30-bulk-pdf-agent-arrnc-hosting-plan.md.

This document supersedes every PDF-agent hosting/storage-deployment statement in the 2026-08-11 Bulk PDF design. It does not supersede that design's upload, security, processing, or domain behavior.

1. Objective and non-goals

All SyRF Bulk PDF processing environments use separate Docker containers on ARRNC API and exact environment-owned children of the Edinburgh Camarades share. Production and staging are persistent, while PR preview slots are bounded and lifecycle-managed.

This planning change does not:

• change runtime code, server configuration, cluster GitOps, credentials, storage, queues, or live containers;
• authorize an ARRNC apply or a Kubernetes deletion;
• merge itself or approve Phase 2;
• promote or alter production.

2. Evidence and precedence

The correction resolves four facts in this order:

1. The owner's later explicit ARRNC-for-all-environments decision is the product authority.
2. The preserved Juniper rewrite corroborates child-scoped production, staging, and preview roots, bounded preview slots, and teardown, but is not copied wholesale and remains untouched.
3. Current repositories define the implementation surface: SyRF has the agent and application authority; server-config has static production ARRNC foundations; arrnc-api-deploy has the gatekeeper transport; cluster-gitops owns the live staging Kubernetes workload.
4. Read-only live evidence proves the stale recovery is active: Argo CD reports staging Healthy/Synced with one Deployment, three containers, ingress, and a bound 20 Gi PVC.

Operational approval of that recovery was not an explicit architectural reversal. Therefore the live workload is preserved as the rollback source until ARRNC staging is proven.

3. Target architecture

flowchart LR
    APP[SyRF API / PM<br/>environment authority] -->|environment queue| MQ[RabbitMQ]
    S3[S3 staged ZIP] -->|presigned GET| A

    subgraph ARRNC[ARRNC API host]
        REG[Config service<br/>bounded registrar]
        P[production agent<br/>persistent, untouched]
        T[staging agent<br/>persistent]
        R[PR-N agent<br/>dynamic]
        CP[production clamd]
        CT[staging clamd]
        CR[PR-N clamd]
        MP[exact child mount<br/>production]
        MT[exact child mount<br/>staging]
        MR[exact child mount<br/>previews/pr-N]
        REG --> P
        REG --> T
        REG --> R
        P --> CP
        T --> CT
        R --> CR
        P --> MP
        T --> MT
        R --> MR
    end

    MQ --> P
    MQ --> T
    MQ --> R
    A[Selected environment agent]
    A --> IIS[Environment-scoped IIS URL]

Only one environment agent receives a given queue. The diagram shows possible slots, not simultaneous consumers for one environment.

4. Repository and ownership boundaries

Repository	Owner	Authoritative responsibilities	Expected implementation surfaces
camaradesuk/syrf	SyRF maintainers	image, runtime configuration, application storage authority, broker naming, workflows, health and E2E harness	src/services/pdf-agent; src/libs/kernel/SyRF.SharedKernel/Settings/BulkPdfStorageSettings.cs; .github/workflows/ci-cd.yml; .github/workflows/pr-preview.yml; .github/workflows/_preview-gitversion.yml; CLAUDE.md; docs
camaradesuk/server-config	ARRNC platform maintainers	users/ACLs, child mounts, credentials, Docker networks, ClamAV, persistent slots, dynamic registrar policy, reconciliation, verification	vars/projects.yml; vars/portal-env-overrides.yaml; deployments/arrnc-api.yml; roles/cifs_mounts; roles/clamav; roles/docker_volumes; roles/gatekeeper; roles/verify; config-service; filter_plugins/tests/test_syrf_pdf_agent_hosting.py
camaradesuk/arrnc-api-deploy	ARRNC gatekeeper maintainers	constrained deploy/apply transport and installer contract	scripts/container-web-deploy; scripts/container-web-apply; scripts/install-gatekeeper; config/services.conf.example; tests
camaradesuk/cluster-gitops	GKE platform maintainers	reversible drain and retirement of the old staging Application, ingress, and PVC	syrf/environments/staging/pdf-agent; syrf/environments/production/pdf-agent/config.yaml; syrf/services/pdf-agent; argocd/applicationsets/syrf.yaml; argocd/applicationsets/syrf-previews.yaml
University Camarades storage service	storage owner; implementation interface is a preview-activation dependency, not a persistent-staging merge gate	audited per-PR child/allocation, principal, ACL, quota, I/O-limit provisioning, reset/fencing, and exact retirement	scoped provisioning operation or isolated allocation pool; audit and simultaneous bounded-I/O proof

Secrets stay in their existing repository- and host-owned secret systems. Plans, PR descriptions, logs, and proof bundles use secret names and redacted fingerprints only.

5. Storage contract

5.1 Canonical layout

The application value object remains the canonical naming authority:

Runtime	Application root	Host mount	Container output	URL/read contract
Production	production	/srv/data/syrf-pdf/production/output	/app/data/output	/production/
Staging	staging	/srv/data/syrf-pdf/staging/output	/app/data/output	/staging/
Preview N	previews/pr-N	/srv/data/syrf-pdf/previews/pr-N/output	/app/data/output	no public URL; scoped inspection only

Scratch is a private host directory under the environment slot, never a share path and never served by IIS. Production and staging public URLs must end in the same canonical root accepted by BulkPdfStorageSettings; previews use the canonical root only for internal record/path identity.

5.2 Isolation and permissions

Each environment uses a distinct SMB principal, root-owned credential file, child mount, Unix UID/GID mapping, and server-side child ACL. Production, staging, and preview identities cannot list or traverse sibling children. The agent is non-root and receives only the exact output bind and its own scratch.

Preview PR code receives neither credential files nor mount commands. Root-controlled server-config mounts the child before the approved rootless container starts. No container gets CAP_SYS_ADMIN, host PID/network namespaces, privileged mode, the Docker socket, or a writable host root.

The storage owner, not GitHub or the PR container, creates or allocates and retires the remote previews/pr-N child, distinct principal, ACL, quota, and server-side connection/bandwidth/IOPS policy through an audited operation or equivalently isolated pre-provisioned pool. Allocation, reset/fencing, identity rotation, and audit must prevent prior or sibling PR exposure. The registrar reserves the allocation in the same atomic fleet admission transaction and verifies sibling denial and I/O limits before activation. The missing operation is an implementation dependency to deliver before preview activation; it does not change the approved ARRNC/container architecture or block persistent staging work.

ARRNC root mounts the exact remote child and bind-mounts only that child into the disposable rootless container. SMB credentials, sibling mounts, and the common root are absent from the container. Lifecycle tests cover exact source, no-follow path handling, sibling denial, mount loss, ordered container stop/unmount, and exact allocation retirement.

The preview bridge is combined with root-controlled deny-by-default egress enforcement. Only its own scanner and the minimum controlled DNS, presigned-S3 HTTPS, and preview RabbitMQ destinations are allowed. Host gateways, metadata endpoints, ARRNC/UoE internal CIDRs, persistent-environment endpoints, and arbitrary internet egress are denied. PR code cannot change the policy. Both positive routes and denied host/internal routes are activation tests; a bridge without this enforcement is not an isolation boundary. Root-controlled shaping applies per-preview ingress/egress rate and burst ceilings to allowed destinations, including presigned S3. Simultaneous bounded transfers must leave persistent services healthy and the ARRNC uplink within its reserved aggregate budget.

The trust boundary admits only explicitly authorized, same-repository SyRF heads. Each gets its own disposable rootless Docker container on ARRNC with a pinned digest, non-root UID, dropped capabilities, no-new-privileges, read-only root filesystem where supported, seccomp/AppArmor, private network namespace, no Docker socket, no host namespace/device/admin access, and no production/staging mount, credential, route, or service identity. A pull request whose head repository is not exactly the SyRF base repository, including every fork, cannot use this path; neither can a stale, closed, or unapproved same-repository head. VM/microVM isolation is optional future hardening, not an activation prerequisite for this approved workload policy.

Fleet admission atomically reserves CPU, memory, PIDs, block I/O, Docker/runtime disk, scratch, storage quota, ports/routes, network, RabbitMQ, and scanner capacity before creating any slot. The configured concurrency limit derives from measured ARRNC capacity and can exceed one; requests queue without partial state when the safe aggregate budget is unavailable. Bounded write/discard/fsync and simultaneous-preview tests prove the reservations and acceptable persistent-slot storage latency/health. Immediately before a queued request reserves resources, the registrar atomically revalidates the exact repository, open/enabled state, current head SHA, verified digest, and current authorized-maintainer approval; stale requests are discarded without provisioning.

Preview output is generated by code under test and can be malformed even when its approved source is trusted. IIS and every other public/static-serving route therefore deny the preview child. A typed host-side inspect operation accepts only the registered PR, persisted job ID, expected derived relative output path, and idempotency key from a short-lived preview-family harness identity. Root resolves the path no-follow beneath the exact mounted child, enforces byte/rate limits, and passes it to an isolated trusted type/size validator and scanner; neither raw bytes nor storage credentials reach the GitHub runner. The operation returns only a redacted size/hash/scan result, is audit-bound to the preview record, and is revoked before container and child teardown. Public preview-PDF serving requires a future separately reviewed trusted post-write promotion boundary into a different read-only served tree.

Preview input is synthetic-only. A protected base-branch harness outside PR execution uses a protected synthetic identity/API path to create a matching Project and initiate the persisted upload session/job, uploads and acknowledges every approved non-sensitive multipart fixture part, then completes the session so the durable job records the environment, size, hash, and policy required by the claim consumer. A trusted notifier release gate durably holds the exact object/version event across the S3-completion/application-persistence race and releases it only after reading the matching committed Uploaded job. Release and crash recovery are idempotent and cannot lose or publish a held event early. Human upload initiation is disabled by an external preview control. PR code cannot select production, clinical, participant, or other sensitive objects.

5.3 Sentinels and readiness

The operator creates .syrf-storage-ok only on a verified mounted child. Its structured, non-secret content includes schema version, canonical root, expected mount source fingerprint, and environment identity. The agent validates content, not mere presence.

The existing IIS Camarades root must be proven read-only to expose the staging child URL while denying every preview child plus directory listing, scratch, siblings, and unscanned content. If the existing mapping does not provide that exact behavior, activation stops for a separately reviewed, declarative IIS-owner change; an ad-hoc static server in the agent is not a fallback.

Readiness fails closed unless all of these agree:

• configured application root and independent runtime/active environment;
• expected direct child mount source;
• sentinel schema and environment;
• effective UID/GID write and rename/fsync capability;
• canonical no-follow resolution beneath the child;
• minimum free bytes and quota headroom.

The mount unit restarts only the matching environment container after recovery. A missing or stale mount never falls through to the local directory underneath it.

5.4 Capacity, backup, and retention

• Production: no Phase 2 staging change may alter its mount, credentials, files, slot, queue, image, or flag. The ARRNC host backup excludes this mount; Gate C therefore requires the remote share owner to attest its actual backup/retention and a restore proof before any promotion.
• Staging: non-production and excluded from production backup. Delivered bytes remain while any durable job, Study, report, or downstream reference can generate or consume their URL. The 30-day policy applies only to proven unreferenced/orphan test data after coordinated application-state retirement; there is no blanket age-delete.
• Preview: no backup; teardown after closure/disablement, hourly reconciliation, and a 24-hour convergence SLO.
• Quotas: server-enforced per child. The implementation PR records proposed byte/inode limits from measured workload evidence and ARRNC free capacity; activation fails if limits and alerts are absent.
• Preview concurrency: a configurable value computed from measured CPU, memory, PID, block-I/O, runtime-disk, scratch, storage-quota, port/route, network, RabbitMQ, and scanner budgets. The registrar may admit multiple previews only when its atomic fleet ledger can reserve every required resource; otherwise it queues the request without partial state.

No cleanup process may receive production-root delete authority. Preview cleanup derives one canonical previews/pr-N child, refuses symlinks/reparse points, and records what it removed.

6. Container and runtime contract

6.1 Images

Staging and production select the same release image by GHCR digest. Preview N selects a PR-specific digest built from the same Dockerfile. A tag can be input to resolution but never the reconciled identity. Provenance binds repository camaradesuk/syrf, commit SHA, workflow, and PR.

Protected activation reads the registry manifest's compressed total and verifies a signed build/verifier attestation of unpacked size before host pull. It enforces reviewed ceilings plus Docker data-root headroom for pull, unpack, and rollback margin. Teardown removes only preview image references/layers proven unused by every live and persistent slot. Reconciliation finds leaked preview layers and headroom drift; ambiguous ownership quarantines and alerts instead of invoking a broad prune.

6.2 ClamAV

Every slot uses the same ClamAV 1.4 image/configuration and limits required for the agent's maximum entry size. Staging and production use separate networks and scanner containers. A preview uses a per-preview scanner and network so one approved workload cannot reach a persistent scanner or consume capacity reserved for another slot. clamd has no published host port.

A trusted updater with a separate network identity may reach only a pinned internal signature mirror. It verifies and publishes a read-only signature set to the scanner. The PR-controlled agent receives no mirror route; refresh, stale-signature fail-closed, and mirror-failure tests prove that updates do not widen preview egress.

Readiness proves the socket, engine version, current signature database age, and EICAR detection. Signature freshness has an alert threshold and fails new work closed once the approved maximum age is exceeded.

6.3 RabbitMQ

Each environment has a distinct virtual host, queue, user, and least-privilege permissions. Preview credentials are minted for one PR and revoked during teardown. Broker credentials are environment-scoped secrets delivered only to the matching slot.

For approved preview clients, broker policy additionally enforces per-preview connection/channel, message-size, publish-rate, queue-length/bytes, and storage limits, with shared-broker free-space alarms. Activation includes simultaneous-client limit tests. If the shared RabbitMQ service cannot isolate aggregate preview limits from staging and production, requests queue or previews use a separately reviewed dedicated broker.

The agent uses its existing durable retries, journal, MassTransit error transport, and environment-root messages. Queue health records ready, unacknowledged, consumer count, oldest age, and dead/error transport depth. Readiness requires exactly one intended consumer when enabled and zero while paused.

6.4 Resource and lifecycle controls

server-config declares CPU, memory, PID, scratch/quota, restart, and log-rotation limits. Preview containers additionally receive host-enforced block-device bandwidth/IOPS ceilings covering their writable runtime data and scratch, with persistent-slot health included in simultaneous bounded-I/O proof. All environments use the same health vocabulary:

• storage-authority and sentinel valid;
• clamd ready and signatures fresh;
• broker connected and exact queue identity;
• consumer state matches desired paused/enabled state;
• scratch journal recoverable and free space above threshold.

Stop is ordered: close new-session admission while existing multipart sessions retain their sign/acknowledge/heartbeat/complete/cancel/abandon paths, prove those sessions quiescent, then pause/fence the consumer, wait for or safely abandon the active claim, flush the journal, and stop. Restart replays idempotently from durable broker and journal state. Rollback pins the previous verified digest; it never starts a second consumer.

7. Declarative ARRNC control plane

7.1 Persistent records

server-config adds a static staging project, mount, env scope, network/scanner, verification, and deployment-manifest entry beside the existing production foundation. Both remain gated while implementation and dry-run proof are assembled. Production remains managed/paused exactly as its pre-migration state requires; it is not adopted incidentally.

7.2 Preview family registrar

The config service gains a typed preview-family operation rather than accepting free-form DeploymentEntry fields. Input is limited to:

• canonical positive PR number;
• verified SyRF PDF-agent image digest and source commit;
• requested desired state create, pause, resume, or retire;
• authenticated repository/workflow identity and idempotency key.

It derives container name, app/environment, storage root, mount, network, scanner, broker scope, secret scope, resource profile, and retention record from root-controlled templates. Structural validation checks the derived record both before commit and before reconcile.

Create first executes one atomic fleet-admission transaction that reserves CPU, memory, PIDs, block I/O, Docker/runtime disk, scratch, storage quota/allocation, ports/routes, network, broker, and scanner capacity. The configured concurrency ceiling is computed from measured budgets and may permit multiple simultaneous previews; a request queues with no partial resources when the aggregate safe budget is unavailable. Create and retire are auditable state transitions. A per-PR lock prevents duplicate operations while the fleet ledger serializes competing reservations. Reconciliation compares open explicitly enabled trusted PRs with desired records, repairs or quarantines orphaned partial state, and removes neither persistent slot. A failed reconciliation alerts but does not widen a path or secret scope.

7.3 Trust and provenance

Same-repository branch code is eligible only after explicit approval by an authorized maintainer bound to its exact head and digest. The branch build job may publish only the image digest and attestation and receives no ARRNC, storage, broker, registrar, or protected-environment credential. A protected workflow sourced only from the base branch performs activation with a short-lived preview-family-only identity. It does not check out or execute PR code and pins every privileged action/reusable workflow dependency. A fork cannot receive secrets or ARRNC execution. Manual promotion of a fork result, if ever needed, is a separate admin-reviewed workflow that rebuilds the exact commit in a trusted context.

The registrar verifies the digest's repository/commit attestation, the PR is open, the enabling approval/configuration is recorded against the exact current head SHA and digest, every required fleet resource can be atomically reserved, and no slot for N points at another digest. Every synchronize event invalidates activation approval, and the SHA/digest binding is rechecked immediately before registration. Queued requests repeat the full repository, open/enabled, head/digest, and approval check immediately before reservation and provisioning. Mutable tag-only requests fail.

8. Staging coexistence and cutover

8.1 Build without consumption

ARRNC staging is first applied with the Bulk feature disabled and agent consumption paused. Host proof covers mount/ACL/sentinel, container limits, ClamAV/EICAR, broker authentication without a consumer, public URL isolation, logs/metrics, and stopped/absent rollback. Previous-digest rollback is additionally required only when a prior ARRNC digest has independently proven paused startup with zero queue subscriptions.

The Kubernetes staging workload remains healthy and authoritative during this work.

8.2 Single-owner handover

At Gate B:

1. close only new-session admission in staging while accepted multipart sessions retain lifecycle endpoints, then prove those sessions quiescent;
2. pause new notifier publication, capture a durable S3-object/outbox inventory, and require every accepted job terminal; Uploaded or another known nonterminal state is not sufficient;
3. drain the Kubernetes queue/claims/journal and create an initial manifest of PVC output, transferring it through a reviewed controlled path into the ARRNC staging child, and prove checksums, canonical paths, stored-link compatibility, and candidate IIS reads; no overwrite or unexplained difference is allowed;
4. switch the reviewed staging public-base/routing configuration to the ARRNC child while the old Kubernetes static server still runs, then prove old and new references resolve from ARRNC;
5. merge/sync a cluster-gitops drain change that scales only the staging PDF-agent Deployment to zero while retaining Application, ingress, PVC, and configuration;
6. with the old consumer fenced, perform a final delta copy, prove no post-fence PVC writes, and reconcile every S3 object/outbox entry with terminal state or a preserved durable queue event;
7. unpause ARRNC staging, prove consumer count exactly one, and resume notifier publication;
8. reopen staging new-session admission and run authenticated end-to-end tests.

No dual-consumer soak is permitted. If any proof is ambiguous, intake stays off and the sequence stops.

8.3 Authenticated proof

The proof bundle is redacted and contains:

• ARRNC desired record, reconciled digest, container limits, and health;
• exact child mount/source, sentinel, ACL denial to siblings, quota, and public-read mapping;
• normal authenticated two-file upload through UI/API, S3, notifier, RabbitMQ, agent, PM, SignalR, CSV, Study link, and IIS read;
• EICAR terminal Infected result with no delivered bytes;
• duplicate/redelivery and restart replay with one final outcome and no duplicate writes;
• clamd loss, RabbitMQ loss, mount loss, process kill, full-disk threshold, and recovery;
• queue drain, consumer ownership, error transport, logs/metrics/alerts, and rollback rehearsal.

9. Kubernetes retirement

ARRNC acceptance begins a soak while the Kubernetes Deployment remains at zero and the PVC stays bound. The first retirement PR disables/removes ingress and runtime ownership but retains a restorable storage artifact. Before PVC removal:

• enumerate and checksum the PVC content;
• explain every difference from ARRNC staging output;
• create and restore-test a VolumeSnapshot or equivalent controlled archive;
• record retention owner, expiry, and restore command;
• prove queues, claims, and journals remain owned by ARRNC.

The final cluster-gitops cleanup removes the Application and obsolete PDF-agent service values, then the PVC only after the retention gate. Reverting Git plus restoring the tested snapshot must recreate the old shape without activating it. While ARRNC is fenced, all output written since snapshot capture is inventory-checked and copied back with checksum/collision proof before the old public route or consumer is enabled. Imperative kubectl delete and orphan-prune shortcuts are forbidden.

10. Preview lifecycle

Activation sequence:

1. approved same-repository PR opt-in and authorized-maintainer approval are bound to the exact current head SHA and verified digest;
2. atomically reserve CPU, memory, PIDs, I/O, runtime disk, scratch, storage quota/allocation, ports/routes, network, RabbitMQ, and scanner capacity from the fleet ledger, or queue without partial state;
3. provision the exact rootless container, broker scope, network/scanner, child mount, quota, sentinel, and derived record for that PR;
4. reconcile container paused;
5. prove isolation/readiness;
6. expose preview Bulk configuration and enable its consumer;
7. run a small authenticated clean/EICAR smoke.

Teardown sequence:

1. remove preview intake configuration;
2. fence consumer and drain/quarantine its queue and claims;
3. stop/remove exact agent and scanner;
4. revoke exact broker and secret scopes;
5. unmount the ordinary agent's exact child;
6. give the root-controlled cleanup worker a one-use exact-child capability, prove its view is the remote previews/pr-N child rather than the local mount underlay, no-follow delete that child, and remove the cleanup view; if this proof is unavailable, quarantine and alert instead;
7. revoke the exact output-inspection capability, remove the desired record and disposable rootless container, per-preview network/routes/ports, writable runtime state, and only image layers proven unused by another slot;
8. prove process/runtime/network/storage/credential absence, then atomically release only the proven-absent parts of the fleet reservation and publish durable absence proof. Any quarantined resource remains charged to the owning reservation, or an atomic quarantine debit against the same aggregate budget, until reclamation is proved.

Close, label/config removal, and reconciliation all call the same idempotent state machine. Failed cleanup is visible and retried; it never deletes another PR or a persistent environment.

11. Observability and rollback

Dashboards/alerts distinguish environment and PR without logging secrets or filenames. Minimum signals are desired/reconciled digest, container state/restarts, CPU/RAM/PIDs, scratch/share free space and quota, mount/sentinel, clamd readiness/signature age, RabbitMQ consumer/ready/unacked/ error depth, current claim age, processing duration/outcome, and preview teardown age.

Rollback gates:

• before Gate B: stop ARRNC staging; Kubernetes remains unchanged;
• after Kubernetes scale-to-zero but before retirement: fence ARRNC and drain, reconcile any ARRNC-created output back to the retained PVC with checksum and collision proof, scale Kubernetes back to one and prove its static server ready, restore the prior public-base/routing configuration, prove sole consumer and old/new link continuity, then restore intake;
• after Application removal but before PVC expiry: fence ARRNC, revert Git, restore/rebind the tested storage artifact, checksummed-copy all post-artifact ARRNC output into it, and prove current-link continuity before enabling the old route or consumer;
• image regression: intake off, drain/fence, pin prior digest, prove sole consumer, intake on;
• storage isolation/provenance failure: stop immediately; there is no permissive fallback.

12. Approval model

Gate A approves the architecture and numbered plan only. Gate B separately approves the live staging drain, ARRNC consumer activation, and later Kubernetes retirement after dry-run proof is reviewed. Production has an additional Gate C with a separate PR set and explicit approval.

Merging implementation code/config does not imply an environment apply. Applying staging does not imply Kubernetes deletion. Staging acceptance does not imply production promotion.