Skip to content

ADR-012: Bulk PDF upload — hybrid pipeline with on-prem processing agent

Hosting correction (2026-08-30): ADR-015 supersedes Decision 2 and every staging/preview Kubernetes or PVC hosting statement in this record. Those statements are historical context only and must not be implemented. The corrected target is a separate production, staging, and dynamic PR-preview Docker container on ARRNC API, each with an exact child-scoped share mount.

Status

In-Review. The architecture, including the multipart reliability refinements and provisional timing defaults, was approved by the product owner on 2026-08-11; that approval is not approval to merge a rollout wave or enable production. Supersedes the unmerged branch document ADR-008-bulk-pdf-s3-only.md from PR #2373 (which collided with main's ADR-008 numbering and was contradicted by that branch's own implementation).

Context

Bulk PDF upload (epic #2223) lets project admins deliver full-text PDFs themselves instead of emailing them to the SyRF team. Final PDF storage is the Edinburgh IIS web server (ecrf1.clinicaltrials.ed.ac.uk/camarades/, ~700 GB of PDFs already served from there), reachable only from inside the university network — nothing in AWS or GKE can write to it.

The feature's storage architecture flip-flopped across 2026: hybrid (S3 staging + on-prem final, discussion #2094, Oct 2025) → S3-only (branch ADR-008, Jan 2026) → hybrid again (decision C-2, Mar 2026, recorded only as a Teams confirmation). The contradiction was never resolved in a merged decision record. This ADR settles it.

Decision

  1. Final storage stays on the Edinburgh IIS share. S3 (syrfapp-uploads*) is staging only; completed ZIPs expire after 7 days. Incomplete multipart uploads have a separate lifecycle cleanup rule as a safety net.
  2. One processing locus, on-prem. A new .NET worker (PDF agent) performs download, ClamAV scanning (clamd companion container), validation, extraction, conflict handling, and the final filesystem write. Per ADR-015, production, staging, and dynamic PR previews use separate Docker containers on arrnc-api with exact environment-child mounts. No PDF agent runs in Kubernetes.
  3. The Lambda stays thin. The existing s3-notifier gains a BulkPdfUpload kind: it validates metadata, generates a 12 h presigned GET, and sends one MassTransit command to the agent's queue. No scanning or extraction in Lambda.
  4. Domain authority stays in the PM service. The agent reports per-file outcomes; PM performs study matching, marks studies, computes counts, and generates the CSV report in a single-writer consumer.
  5. The browser builds the ZIP; the server owns its multipart upload session. Users select a folder of PDFs; the client validates (PDFs only), previews matches against the search, and zips in a web worker (STORE). It initiates a PM-backed S3 multipart session bound to exact size and whole-object SHA-256, then uploads parts directly to S3 using short-lived presigned UploadPart URLs whose SigV4 signature binds the exact part Content-Length and SHA-256. The browser uploads an exact known-length Blob.slice; its user agent supplies the forbidden Content-Length header, so that header is not returned as a JavaScript-settable RequiredHeaders value. The API does not proxy bytes.
  6. No cloud credentials or inbound connections on university hosts. The agent pulls S3 via presigned URLs and AMQP outbound only.
  7. PM owns upload liveness and completion. A configurable 20-minute lease is renewed by a 60-second heartbeat and after the backend verifies a completed part; part URLs have a separate 5-minute request-start validity. The API also advertises a configurable 10-minute maximum client UploadPart request duration; it is not the URL-expiry window. Only the backend calls CompleteMultipartUpload, after an atomic Uploading → Completing transition and exact manifest verification. That transition atomically persists a durable completion-operation claim shared by the original API request and PM recovery. The claim is three minutes by default, the server-owned storage operation is bounded to two minutes, and every storage side effect and result mutation revalidates exact live ownership. PM excludes a live claim, takes over an expired one atomically, and applies the existing 60-second recovery grace to legacy Completing rows with no claim. A final claimed object HEAD is mandatory before PM may enter Aborting. These accepted defaults remain subject to the pre-launch slow-network/background-device timing gate.
  8. Terminal state converges monotonically. Abandon, cancel, and lease expiry atomically enter Aborting; no heartbeat, stale callback, reconnect, or late successful part may restore Uploading or trigger completion. The API persists the latest issued part-URL expiry before returning that signature. After entering Aborting, PM promptly issues AbortMultipartUpload, but cannot count an absence observation until max(AbortStartedAt, LatestSignedPartUrlExpiresAt) + MaximumUploadPartRequestDuration. With the accepted five-minute request-start validity and ten-minute maximum active request, the worst-case initial quiescence boundary is therefore 15 minutes, not a fixed post-abort grace. PM repeatedly aborts, lists parts, and checks the staging object. It requires two joint empty/NoSuchUpload MPU and object-absence observations at least one 30-second sweep interval apart; any non-empty MPU or present object resets that proof. A present object is deleted and must be re-observed absent on a later pass. Only then may PM record Abandoned/Cancelled and release active capacity. Only legacy jobs that truly have no MPU may terminalize on direct authoritative absence proof. Each cleanup pass first atomically acquires a durable, separate exact-token abort-sweep lease (three-minute default) and is bounded to a shorter storage timeout (two-minute default); active claims are not selected, expired claims are safely taken over, and every storage call and proof mutation revalidates the unexpired claim. Normal passes release promptly; process/host cancellation safely leaves the claim for expiry takeover rather than delaying shutdown. The abort-sweep lease is not reused as the completion-operation lease. Conversely, a winning Completing/Uploaded transition rejects late abandon. Retry v1 creates a fresh independent session; it is not resumability.
  9. Processing requires a PM claim before side effects. After object completion, the agent's first operation is an atomic PM claim of the exact upload/size/hash binding. Only an accepted claim returns the opaque token required on progress and finalize messages; download, scratch, sentinel, scan, and copy all happen after the claim.
  10. The browser renders truthful, reconciled state. Uploading means an UploadPart request or valid bounded retry is actually active. Recoverable loss is shown as retrying; terminal local failure stops animation, queues idempotent abandonment durably, and offers fresh Retry. PM state is reconciled on reconnect/reload through SignalR plus heartbeat/status and bounded polling fallback. A terminal server state cancels local work; a completion winner is rendered as completion.
  11. Bulk storage authority is explicitly environment-owned. BulkPdfStorage:EnvironmentRoot has no default and accepts only the canonical values production, staging, or previews/pr-N (a positive, non-zero-padded PR number). A preview root must exactly match SyrfSettings:PrNumber; every root must also match the independently rendered RuntimeEnvironment and ActiveEnvironment exactly (production, staging, or preview). This catches a staging workload configured with the production root (and the converse) instead of treating the root configuration as self-authenticating. The API resolves this setting only after the default-off bulkPdfUpload gate is open, so disabled deployments remain healthy without it. Once the gate is on, absent, malformed, or deployment-mismatched authority returns typed 503 rather than falling back to a shared or inferred root. New sessions persist the canonical root immutably in the PM job and in the S3 object/MPU metadata. Status reads do not grant storage authority; every signing, heartbeat, completion, abandonment, retry, or other storage-mutating operation requires the current root to exactly equal the job root. A legacy/rootless job, or a job from another root, is rejected fail-closed (409) for mutation and is never adopted by a later deployment.

The accepted environment root is also carried unchanged on every agent progress and finalize command. The PM consumer requires an exact three-way match between message, persisted job, and resolved local authority before any state, CSV, or Study effect. A deterministic authority rejection faults to the MassTransit error transport without entering either retry policy; silently acknowledging a one-way command on the wrong deployment would lose it. Retryable aggregate races and transient Mongo/network failures use five bounded 200 ms immediate retries followed by durable publish-scheduler redelivery after 5 seconds, 30 seconds, 2 minutes, 10 minutes and 30 minutes. Exhaustion reaches error transport; authority faults and ordinary defects never enter either retry policy. This is defense in depth, not broker routing: activation also requires environment-specific RabbitMQ queue routing or namespaces so competing consumers from another environment cannot take a command intended for the local PM instance. The PDF-agent publisher must be deployed with this contract before any upload is activated (or zero in-flight legacy messages must be proven), because the fail-closed PM consumer deliberately error-transports a follow-up whose environment authority is absent.

The always-running sweeper may converge only positively stale rootless records without storage authority. Queued and unmarked Expired records are immediately eligible. Uploading is eligible only after its persisted lease expires. Aborting is eligible only after any durable abort-sweep claim expires and the maximum part-request duration has elapsed from the later of abort start (falling back to creation) or the latest signed-part-URL expiry. Completing is eligible after an exact completion claim pair expires; a missing/partial legacy pair instead waits the configured completion-recovery grace from completion start (falling back to creation). Uploaded is eligible only when both processing-claim fields are absent and the same recovery grace has elapsed from upload completion (falling back to completion start or creation). Each accepted retirement moves the job to non-deletable Expired with a positive RetiredWithoutStorageAuthorityAt marker. Only that explicit rootless marker releases active capacity; rooted or unmarked Expired records retain capacity until ordinary storage-cleanup proof reaches Abandoned or Cancelled. Retirement never claims, aborts, deletes, reads, or otherwise touches storage, and a sweeper lacking exact local deployment identity performs no retirement or storage work.

CSV report reads are storage access and therefore require the loaded job's root to match the local authority before the file stream is opened. Study PDF path/link reads remain an activation dependency: the availability/provenance work must prove that a new Bulk-owned PDF physically exists beneath the local environment root before exposing it. Existing/manual legacy paths are not adopted or reclassified by this release.

This is an application authority contract, not evidence that Kubernetes mounts, CIFS export permissions, static serving, or S3 prefixes are isolated. Those controls require focused server-config/GitOps/hosting work and independent validation. Until that work exists, the feature remains default-off; this decision neither creates directories nor authorizes a deployment or storage cleanup. 12. Bulk PDF path keys use a pinned cross-runtime Unicode contract. The browser preview, PM finalizer/link builder, and PDF agent all apply NFC → the narrow existing typographic punctuation fold → Unicode 16.0.0 default full case folding → NFC → existing separator and marker rules. The checked source is PdfPathNormalization/CaseFolding/CaseFolding-16.0.0.txt, SHA-256 6f1f9c588eb4a5c718d9e8f93b782685e5c7fec872cf05e8e6878053599e09bb; the checked generator emits one JSON mapping consumed by both C# and TypeScript. It selects the Unicode F mapping when present, otherwise C, and excludes S and locale-specific Turkic T mappings. Runtime ToLowerInvariant()/toLowerCase() are not an acceptable substitute. The full fold remains only a matching/collision key: path traversal, Windows portability, component byte budget and canonical destination-path validation remain independent, fail-closed safeguards applied after normalization. Updating Unicode data requires a separately reviewed mapping version, checksum, generator run, shared vectors, C#/web/agent validation, and explicit compatibility review. 12. An upload is bound to the exact Study PDF-path set that was previewed. The API returns the path list with protocol version study-pdf-path-set-v1 and a server-computed SHA-256 fingerprint. Initiate and fresh Retry must present that exact binding; a changed set is rejected with typed study_pdf_paths_changed before storage initiation. The binding hashes strict UTF-8, length-prefixed tuples of Study ID and the exact raw PdfRelativePath, ordered by Study ID. PM recomputes it at finalization before report or Study writes, then applies each delivery marker with an exact project/search/study/raw-path compare-and-set and verifies the marker postcondition. Each string is prefixed by its four-byte unsigned big-endian UTF-8 byte length; the protocol version is the first field and Study IDs use lowercase D format. Drift fails the job truthfully and requires a fresh preview and Retry; it never links delivered bytes to a Study path the user did not preview. Jobs persisted before the binding fields existed remain readable and retain legacy finalization during rolling deployment, but every newly initiated session requires the versioned binding.

Unicode path-key consumers and rollout boundary

The contract is deliberately one matching key, not an alternative filesystem path rule. Its consumers and boundaries are:

Layer Consumers Boundary protected by the same key
Browser pdf-path-normalization.ts, folder-selection.ts, match-preview.ts selected folder entry vs the search's Study.PdfRelativePath, preview collision/portability result
Shared kernel PdfPathNormalizer, PdfPathCollisionDetector, PdfPathPortability one canonical match key; collision and Windows/CIFS validation remain fail-closed
PM BulkPdfUploadFinalizeConsumer, PdfLinkBuilder agent outcome vs study match; immutable delivered-path link validation and CSV normalized_path
PDF agent ZipEntryPlanner raw ZIP entry, pre-write collision detection, path portability and canonical destination relative path

The artifact foundation must land before the web mirror can import it. The web change is a required dependent change: replace its current runtime toLowerCase() step with the shared JSON lookup and a second NFC operation, with no locale-sensitive alternative. The agent inherits the shared-kernel implementation once rebuilt against it. The normalizer is feature-path scoped and Bulk PDF remains default-off; nevertheless, no Bulk activation is permitted until the browser, PM and agent all run the same checked artifact and the shared vectors pass in each relevant suite.

Consequences

Positive

  • Zero broken links: a study's PDF link activates only after the file is verifiably on disk.
  • One place to debug processing; no cross-cloud two-hop pipeline (the #2373 shape).
  • No new AWS infrastructure beyond a chart-declared lifecycle rule; no clamd-in-cloud.
  • Preview/staging environments run the identical ARRNC Docker contract, so the on-prem production shape is E2E-testable before production promotion.
  • ClamAV signatures stay current via freshclam without image rebuilds.
  • A killed browser, powered-off device, or severed connection cannot leave an upload Pending on the assumption that the original non-resumable transfer may spontaneously continue.

Negative / accepted

  • Production depends on a single manually-applied host (arrnc-api) and a CIFS mount; mitigated by fail-closed sentinel checks, durable queueing while the agent is down, and server-config's recovery/verify roles.
  • Unscanned bytes touch the agent's isolated container before scanning: the ZIP is downloaded and extracted into container-scratch space, and every extracted file is scanned there before anything is copied to the share or any served location. The security boundary is "nothing unscanned ever reaches final storage", not "nothing is extracted pre-scan" — scratch is private to the agent container (same trust boundary as scanning inside a Lambda).
  • AMQP to rabbitmq.camarades.net:5672 is currently plaintext over the internet, and this applies to both out-of-cluster clients: the existing production Lambda and the new PDF agent on arrnc-api. A separate hardening PR adds AMQPS; both flip their URIs when it lands.
  • The bulk layout (projects/{p}/searches/{s}/pdfs/…) coexists with the legacy flat layout; link generation branches on Study.BulkPdfDeliveredAt.
  • Multipart upload introduces a PM sweeper and state/storage reconciliation path. Capacity is deliberately retained through the derived latest-signed-URL quiescence boundary and two-observation cleanup proof; the storage lifecycle rule is only a fallback, so transient S3 errors do not manufacture a terminal state.
  • The accepted timing defaults trade prompt interruption detection against slow links and browser throttling. Staging must validate slow 1 GB upload, prove each exact 16 MiB part completes safely inside the advertised 600-second request timeout, and exercise transient loss, heartbeat jitter, and device sleep/background behaviour before production enablement; unsafe evidence requires an explicit narrow timing decision rather than silent retuning.

Alternatives considered

  • S3-only storage, no agent (branch ADR-008; revisited 2026-08-11 as "GKE worker + S3 final"): cleanest platform-native end-state, but changes the PDF serving model and contradicts the standing storage decision; rejected consciously.
  • Keep #2373's two-hop pipeline (scan-in-Lambda → safe bucket → agent): requires clamd reachable from AWS (new always-on infra), doubles S3 I/O, and splits processing across a cloud/on-prem boundary; dominated once an on-prem writer exists anyway.
  • Direct upload to an ingest service on arrnc-api (no S3): loses durable staging, presigned upload UX, and per-PR preview isolation; couples availability to one host.
  • One presigned PUT plus read-time/24-hour expiry: the server cannot know promptly that the browser's non-resumable request is irrecoverably severed, cannot make continuation impossible, and may strand active capacity. Rejected in favour of renewable server-owned multipart state.
  • Browser-owned multipart completion: allows a stale client to race terminal abandonment and publish an object after PM has ended the session. Rejected; completion is backend-only.
  • Resume/adopt prior parts on Retry: adds cross-session trust and ownership complexity. Rejected for v1; Retry is explicit and fresh after authoritative cleanup.

References

Design: docs/superpowers/plans/2026-08-11-bulk-pdf-upload-v2-design.md · Plan: docs/superpowers/plans/2026-08-11-bulk-pdf-upload-v2-plan.md · Epic #2223 · discussions #2093/#2094/#2095 · superseded attempt PR #2373 · ADR-010 (s3-notifier IAM via ACK) and its 2026-08-11 audit.