CAS routes

Every route openxet serve answers, with its credential, request, answer and every status it can return. The routes are the Xet CAS API (https://huggingface.co/docs/xet/api) as stock clients call it; where the spec and hf_xet 1.7.0 differ, openxet serves both, as recorded in S10. This page is written from the code in crates/cas/src; the route list is served() in crates/cas/src/lib.rs, and a test fails if a route is served without being in the credential, scoping and echo test matrices (see security tests).

The Hub API (/api/...), Git and LFS routes arrive in M2.

Conventions

• Origin root. The CAS is served at the root of server.public_url (config); clients append /v1/... to it.
• Credentials. Authorization: Bearer <token>, a static token or a CAS token (config). Read means any valid token; write means a token with mode = "write". A token only ever sees its own scope's files. The probes need no credential, and the transfer route takes its signed URL as the credential instead.
• Hashes in paths are exactly 64 lowercase hex characters, in Xet byte order. {prefix} is default; the dedup route also takes default-merkledb.
• Order of checks. The path is checked before the credential, and both before any body is read. A refused upload's body is read and discarded first, so the client sees the answer, not a reset connection; a body declared over the route's limit, or sent with Expect: 100-continue, is not read, and the answer closes the connection.
• X-Request-Id is on every answer, success or error. The server assigns it and logs the request under it; a client-sent one is ignored.
• Cache-Control: no-store is on every answer of the routes marked private below, errors included: their answers depend on the caller's credential, so no cache or proxy may serve one caller's answer to another. /v2/shards streams are no-cache; the probes and POST /v1/telemetry send no Cache-Control.
• HEAD on a GET route runs the GET route's checks and answers without a body, except on the transfer route, where it is 405.
• Bodies are never compressed.

Errors

Every error is JSON with Content-Type: application/json:

{"schema":"openxet.error.v1","code":"unknown_xorb","message":"xorb 1f0c… is not stored on this server. If the server was reset, …","hint":"Upload every xorb before the shard that references it, …"}

code is stable, for scripts and agents; message says what happened, and hint what to do next. A 5xx never carries its cause: give the operator the X-Request-Id, under which the server logged it.

These answers can come from any route:

Routes

Any other path is 404 not_found. That includes the xorb upload-grant route that pre-release hf_xet builds probe: they fall back to POST /v1/xorbs on its 404. Grants stay unserved until a stable client ships them (design §5.5).

GET /healthz, GET /readyz

No credential: orchestrators call them unauthenticated. Each answers a fixed body that names nothing stored.

• /healthz: 200 {"status":"ok"} as soon as the server listens.
• /readyz: 200 {"status":"ready"} once the index and the bucket are open. Before that it answers 503 not_ready with Retry-After: 1, as does every other path, and closes the connection.

POST /v1/xorbs/default/{hash}

Upload one xorb: its chunks as stock clients serialize them, with or without a footer (https://huggingface.co/docs/xet/xorb). The body is validated, and its computed xorb hash must be {hash}. Bytes are stored before the index names them. Uploading again is safe.

200 {"was_inserted": true} if the xorb is new to the caller's scope, {"was_inserted": false} if the scope already had it.

Each xorb_* message names the rule and the chunk it was found in. Clients don't retry a 400.

HEAD /v1/xorbs/default/{hash}

200 with the stored size as Content-Length and no body, or 404 xorb_not_found if the xorb isn't visible to the caller's scope (with dedup_visibility = "per-scope", another scope's xorb reads as missing; config).

POST /v2/shards

Register the files of one upload shard (https://huggingface.co/docs/xet/api#5-streaming-shard-upload). The shard is parsed, checked against the verification limits, verified against the stored chunk hashes of every xorb it references, stored, then its files are registered to the caller's scope. Uploading again is safe.

Before the body is read, a refusal is a status: the credential errors, and 413 shard_too_large for a body over 64 MiB. Once the body is read, the answer is 200 with Content-Type: application/x-ndjson and Cache-Control: no-cache, and every outcome is a line of the stream:

{"type":"validating","verified":0,"total":0}
{"type":"validating","verified":0,"total":3}
{"type":"validating","verified":3,"total":3}
{"type":"committing","stage":"uploading"}
{"type":"committing","stage":"syncing"}
{"type":"result","result":1}

While work runs, the last event repeats every cas.heartbeat_interval (5s by default; config), since hf_xet fails a stream that is silent for 300 s. Exactly one terminal event, result or error, ends the stream.

An error event carries the failure as a message, since the status is already sent; on /v2/shards that message is the only server text the user sees:

• The shard's own fault (shard_malformed, shard_verification_failed or unknown_xorb below): its message, with retryable: false.
• A server fault: a generic message with the request id, and retryable: true if a retry may help (the index or bucket was unavailable), false for a bug. A retryable error makes hf_xet resend the whole shard.

A client that disconnects mid-stream doesn't stop the work: the shard is stored and its files registered all the same, and the client's retry gets result: 0.

POST /v1/shards

The same ingest, answered once: 200 {"result": 1} if anything was new to the scope, {"result": 0} if not. Failures are statuses:

hf_xet doesn't show a /v1/shards error body, so each rejected shard is also logged at warn with its full reason under the request id.

GET /v1/chunks/{prefix}/{hash}

Global dedup: is the chunk with this raw hash stored in a xorb the caller may see? {prefix} is default (what hf_xet sends) or default-merkledb (what the spec names).

200 application/octet-stream: a shard with a footer that offers the xorb holding the chunk and the other xorbs of the newest shard that references it, in that shard's order starting at the matched one, up to 16 MiB of answer. Its chunk hashes are HMAC-keyed with a fresh key per answer that expires in 7 days, so the answer never reveals a raw chunk hash. Which scopes' xorbs are offered follows cas.dedup_visibility (config).

Shards registered before index migration 0002 have no xorb list: a match in one of them offers the matched xorb alone, until the shard is registered again (see the changelog).

A dedup answer must be sent within 30 s plus 1 s per MiB; a client that stalls longer has its connection closed.

GET /v2/reconstructions/{file} and GET /v1/reconstructions/{file}

The plan to download a file registered to the caller's scope, optionally for one Range (https://huggingface.co/docs/xet/api#1-get-file-reconstruction). Range is bytes=start-end (inclusive; what hf_xet sends), bytes=start-, or bytes=-length. A range that ends past the file is clamped to it.

200 application/json. v2:

{
  "offset_into_first_range": 0,
  "terms": [{"hash": "<xorb>", "unpacked_length": 3145728, "range": {"start": 0, "end": 57}}],
  "xorbs": {
    "<xorb>": [{"url": "<signed transfer URL>", "ranges": [{"chunks": {"start": 0, "end": 57}, "bytes": {"start": 0, "end": 3146183}}]}]
  }
}

v1 has the same offset_into_first_range and terms, and fetch_info instead of xorbs: per xorb, a list of {"range": {"start", "end"}, "url", "url_range": {"start", "end"}}. Chunk ranges are end-exclusive, byte ranges end-inclusive. Each fetch entry has one range and its own URL, signed for exactly that range.

The empty file has no satisfiable range: with a Range it answers 416, and without one 200 with no terms ({"offset_into_first_range":0,"terms":[],"xorbs":{}}). The spec says 416; stock clients download the empty file correctly from either (S10 §1). hf_xet always sends a Range, so it gets the 416.

POST /v1/reconstructions and GET /v1/reconstructions?file_id=…

The plans for several files at once (https://huggingface.co/docs/xet/api#6-batch-file-reconstruction). POST takes the spec's JSON list, [{"prefix": "default", "hash": "<file>"}, …], of at most 256 KiB; GET takes file_id=<file> once per file, as xet-client sends it (a URL holds about 880). Both answer alike: 200 application/json,

{"files": {"<file>": [<term>, …]}, "fetch_info": {"<xorb>": [<v1 entry>, …]}}

with terms and entries in the v1 shapes above, each file planned whole. A file named twice is answered once; a file not registered to the scope is left out, the same as one never stored.

HEAD /v1/files/{file}

200 with the file's size as Content-Length and no body, or 404 not_found if it isn't registered to the caller's scope.

GET /v1/transfer/xorbs/{hash}?range={start}-{end}&exp={unix}&sig=…

The data URLs a reconstruction hands out. The signature is the credential: no Authorization is read, as stock clients send none to data URLs. A URL is valid for one xorb, one byte range and until its exp (cas.url_lifetime; config), and only with exactly Range: bytes={start}-{end}.

206 Partial Content, application/octet-stream, with Content-Range: bytes {start}-{end}/{size} and exactly those bytes.

Logs record the route, never the query: sig is a credential.

POST /v1/telemetry

hf_xet reports transfer statistics here. openxet needs a credential, then reads and drops the body: it is never stored or logged (no telemetry unless an operator opts in). 204 No Content.