Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

0015: Run on Postgres and object-store buckets, and sign range-bound presigned URLs ourselves

  • Status: accepted
  • Date: 2026-11-05
  • Scope: index, storage, cas, server, docs
  • Rulings: M3-R2 to M3-R5 (design §17)

Context

M1 and M2 run on SQLite and a local-disk bucket. Design §12 M3 asks for Postgres, S3, GCS and Azure, and presigned downloads; design §5.8 sets the topology: OpenXet nodes scale out, Postgres scales up, with one primary, read replicas for reconstruction and listing reads, and partitioning by hash prefix. The index trait must not assume a single instance, so a sharded backend can come later without touching callers.

ADR 0002 makes every download URL bind one xorb, one byte range and one expiry, and gates direct downloads per pool by a range-binding probe. Its "Open, owned by M3" list asks for the probe against real AWS S3, R2, Garage, GCS and Azure, and for a signing benchmark.

What we checked, and how:

  • object_store's signer can't bind a range. In object_store 0.14.0 (read from the crate source), Signer::signed_url takes a method, a path and an expiry and nothing else (src/signer.rs), and the S3 presigner sets X-Amz-SignedHeaders=host (src/aws/credential.rs). A URL it signs reads the whole object, which ADR 0002 rule 2 forbids.
  • SigV4 and GCS V4 can bind a range. Both query-signing schemes take a list of signed headers (X-Amz-SignedHeaders, X-Goog-SignedHeaders). Spike S1/S2 observed MinIO enforcing host;range and stock hf_xet downloading through such URLs (ADR 0002, Context). Whether AWS S3, R2, Garage and GCS enforce it is not yet observed; ADR 0002's probe decides per pool, never a list.
  • Azure SAS has no field that binds a byte range. A SAS token signs permissions, a time window, a resource and optional IP and protocol limits, and nothing about the request's Range.
  • Registration records rely on create-if-absent (ruling M1-R37). On an object store that is PutMode::Create. An S3-compatible store that accepts If-None-Match: * and overwrites anyway would let two writers' records diverge, so the server must not run on one.
  • SQLite's single writer hid every write race. Two nodes registering one shard at once, or committing to one repository, must both succeed or one win cleanly on Postgres, never answer 500 from a unique violation or a serialization failure.

Decision

  1. Postgres is a second implementation of the existing traits. Index, Repos and GitStore get a Postgres implementation in openxet-index, with its own migrations, and both backends pass one backend-agnostic conformance suite. The schema mirrors SQLite's (hashes as BYTEA, times as BIGINT Unix seconds). xorbs, xorb_layouts, xorb_owners, file_scopes and dedup are PARTITION BY HASH on their hash column with 16 partitions; changing the count is a migration. Chunk tables and layouts stay per-xorb blobs, never per-chunk rows (design §5.8). The registration sequence is a Postgres SEQUENCE.
  2. Races are success, or one clean winner. Inserts of content-addressed rows use ON CONFLICT DO NOTHING and read back, so a lost race is success. The row of the lowest (seq, shard, tenant) (ruling M1-R37) is kept by a conditional upsert. A ref commit locks the repository row (SELECT ... FOR UPDATE) in the transaction that checks the fence (ADR 0004 item 3). A serialization failure or deadlock (40001, 40P01) is retried up to 3 times, then answered 503 index_busy with Retry-After: 1.
  3. Configuration. [index] url takes a sqlite: path or a postgres:// URL (default: the data directory's index.sqlite), max_connections (default 4 × CPUs, bounds 2 to 1024) and an optional read_url for a replica.
  4. A replica never answers "absent" alone. With index.read_url, reconstructions, file heads, listings and dedup lookups read from the replica. A replica answer of "absent" is read again from the primary before anyone is told "absent", so a replica never hides an acknowledged write (§5.9 requirement 7). Writes, Git ref snapshots and every authorization-bearing read (tokens, revocations, limits) use the primary.
  5. Object-store buckets. [storage] url selects the backend: file:///path (M1's local bucket, default {data_dir}/bucket), s3://bucket/prefix, gs://bucket/prefix or az://container/prefix, with object_store's builder options under [storage.options]. The key layout is M1's, under the prefix. Credentials come from the environment, instance identity or a *_file option, never from the config file in plain text. A remote store requires an explicit cas.transfer_key_file and an explicit CAS-token key (ruling M1-R29), since every node behind one endpoint needs the same keys.
  6. A conditional-create probe at start-up. Registration records are written with PutMode::Create. At start-up each node creates probe/{node uuid}/create twice and requires the second call to report that the object exists. A store that fails is refused with storage_unsafe, naming the check and the setting; the server never runs on it. Probe objects live under a reserved prefix that list never returns, are reused rather than deleted, and are swept by M4's GC.
  7. OpenXet signs presigned GETs itself, with Range signed. One small signer per store family: SigV4 query signing with X-Amz-SignedHeaders=host;range for s3:// (AWS S3, R2, MinIO, Garage), and GCS V4 signing with X-Goog-SignedHeaders=host;range for gs:// (service-account key or HMAC key). az:// pools have no presigner and serve through the transfer role. Each signer is tested against its provider's published signing examples.
  8. Direct URLs only after the probe passes, in this process. Each pool runs ADR 0002's range-binding probe at start-up and in openxet config check --probe, against a fixed 64 KiB object at probe/range-binding. Only a pool whose three checks pass serves direct URLs. The outcome is logged, exported as the openxet_storage_direct_downloads gauge and shown by config check. A pool that fails is reported as serving through the transfer role, with the check that failed, and nothing more.
  9. Presigned lifetime. A presigned URL lives min(cas.url_lifetime, credential expiry - 60 s). When that falls under ADR 0002's 30 min floor, the presigner refreshes its credentials first; if it still can't reach the floor, that answer's URLs are transfer URLs.

Consequences

  • Nothing changes on the wire for stock clients except the URLs inside reconstruction answers, which become native presigned URLs on a pool that passed the probe. Every fetch entry still has exactly one range and one URL (ADR 0002 rule 1).
  • Azure pools serve downloads through the transfer role until a SAS can bind a range. Their bytes pass through OpenXet nodes, which then scale with download traffic; the docs say so where Azure is configured.
  • A replica adds one primary read for every "absent" answer. Absent answers are rare on the hot path (a client asks for files it was told exist), so the cost lands on misses, which is where a lagging replica would otherwise be wrong.
  • Every write path must now be correct under concurrent writers on separate connections. The conformance suite's two-handle cases (concurrent_register_same_shard_two_pools, concurrent_put_xorb_two_pools, concurrent_hub_commits_two_pools_one_winner) and lease_epoch_fences_a_stale_holder_across_pools hold both backends to it.
  • We own two signers. They are small, deterministic and checked against published vectors (sigv4_matches_aws_test_vectors, gcs_v4_matches_reference); a provider change that breaks one makes its probe fail, and the pool falls back to the transfer role rather than serving a URL the store won't honour.
  • Tests that enforce the security side: every_direct_url_binds_its_range, direct_urls_only_after_probe_passed, probe_falls_back_on_unbound_signer, presigned_lifetime_capped_by_credential_expiry, start_refuses_store_without_conditional_create, replica_miss_reads_primary and replica_never_used_for_tokens_or_refs (security test map, §5.9 requirements 2, 3 and 7).
  • ADR 0002's open list is carried out by M3 Task 7: each store's probe outcome, with the date and region, goes into docs/reference/object-stores.md, which says "not yet probed" for a store until it is recorded.

Alternatives considered

  • aws-sigv4 from the AWS SDK. It can sign arbitrary headers, but it brings the SDK's dependency tree for one function. One signer per family is about 200 lines with published test vectors, and GCS needs its own anyway. Rejected.
  • object_store's presigner, with unbound URLs. A URL that reads the whole xorb crosses scopes, since xorbs are shared (ADR 0002, alternatives). Rejected by ADR 0002 rule 2.
  • A static list of stores that support range binding. It would trust documentation over behaviour and miss proxies, CDNs and provider changes. Rejected by ADR 0002 rule 3; the probe decides.
  • A distributed key-value store for the index now. Design §5.8 keeps it for later: partitioned Postgres holds the 1 PB sizing, and the traits don't assume one instance, so it can be added without touching callers.
  • Read everything from the replica. It would let a lagging replica answer "absent" for an acknowledged write, and serve stale authorization data such as a revoked token. Rejected.

Read this page as Markdown · Report a docs issue