Operations

Warehouse CRD

A Warehouse declares an upstream source system — the system of record pipelines extract rows from, plus the verified reachability and credential shape needed to reach it. Data in Layer is derived from a warehouse and reconstructible from it. The serving side is the VectorStore; the two sit on opposite sides of the gateway.

Supported Warehouses

Kind	Status	Source
`snowflake`	Shipped	Snowflake databases and tables, key-pair credential
`huggingface`	Shipped	Hugging Face Hub datasets, public or token-gated
`rest`	Shipped	Any paginated JSON HTTP API
`databricks`	Reserved	Schema-reserved; rejected by the operator until implemented
`iceberg`	Reserved	Schema-reserved; rejected by the operator until implemented

Snowflake warehouses hold a key-pair credential. Hugging Face dataset warehouses can be public and credentialless, or can point at a read-token Secret for gated/private datasets. REST warehouses declare a JSON HTTP API — a base URL, optional auth, and a pagination rule — so any paginated JSON API is a source without a bespoke kind.

apiVersion: hevlayer.com/v1alpha1
kind: Warehouse
metadata:
  name: prod-snowflake
  namespace: layer
spec:
  kind: snowflake
  snowflake:
    account: acme-xy12345
    user: SVC_LAYER
    role: SVC_LAYER_ROLE
    warehouse: EXTRACT_WH
    keyPairSecretRef:
      name: snowflake-rsa
    pool:
      size: 5
      timeout: 30s
  verifyInterval: 1h

apiVersion: hevlayer.com/v1alpha1
kind: Warehouse
metadata:
  name: huggingface-hub
  namespace: hev-shop
spec:
  kind: huggingface
  huggingface:
    endpoint: https://huggingface.co
    # tokenSecretRef is optional for public datasets.
    # tokenSecretRef:
    #   name: hf-read          # key: token
  verifyInterval: 1h

apiVersion: hevlayer.com/v1alpha1
kind: Warehouse
metadata:
  name: openfda
  namespace: layer
spec:
  kind: rest
  rest:
    baseUrl: https://api.fda.gov
    # auth is optional — omit for open APIs.
    # auth:
    #   in: query              # query | header
    #   name: api_key          # the param/header carrying the credential
    #   secretRef:
    #     name: openfda-key    # key: token
    rateLimit:
      requestsPerSecond: 4
    verify:
      path: /drug/label.json   # a cheap GET the operator probes
      query:
        limit: "1"
  verifyInterval: 24h

Connection

spec.kind selects the source system. snowflake, huggingface, and rest select a source; databricks and iceberg are reserved by the schema but rejected by the operator until implemented. Exactly one kind block must be present and match spec.kind. verifyInterval (default 1h) sets the probe cadence for any kind.

A warehouse is identity and credential — not a catalog. What to read — a Snowflake database and table, a Hugging Face dataset and split — belongs to the pipeline source; one credential reaches many.

Snowflake

Field	Purpose
`kind`	`snowflake`, `huggingface`, or `rest`. `databricks` and `iceberg` are reserved by the schema but rejected by the operator until implemented.
`snowflake.account`	Snowflake account identifier.
`snowflake.user`	Service user the key pair authenticates.
`snowflake.role`	Optional role assumed on connect.
`snowflake.warehouse`	Snowflake compute warehouse extraction queries run on.
`snowflake.keyPairSecretRef`	Secret in the same namespace holding `private-key.pem` and optional `passphrase`. The credential is never stored in the CRD.
`snowflake.pool`	Connection pool tuning: `size`, `timeout`.
`huggingface.endpoint`	Hub endpoint. Defaults to `https://huggingface.co`; override for Enterprise Hub or a mirror.
`huggingface.tokenSecretRef`	Optional Secret in the same namespace holding `token` for gated/private datasets. Omit for public datasets.
`verifyInterval`	Probe cadence. Defaults to `1h`.

A warehouse is source identity and credential — not a catalog. Which database, schema, table, dataset repo, config, or split to read belongs to the pipeline source; one credential can reach many tables or repos.

REST / HTTP JSON API

A rest warehouse declares a JSON HTTP API. The credential is optional — many public APIs need none, or take a key that only raises a rate limit (omit auth for the anonymous case, the same Verified-with-no-Secret shape as a public Hugging Face dataset).

Field	Purpose
`rest.baseUrl`	Required. API origin. A source’s `request.path` resolves against it.
`rest.auth`	Optional. `in` (`query` or `header`), `name` (the param or header carrying the credential), and `secretRef` (Secret holding `token`). Omit for open APIs.
`rest.rateLimit`	Optional. `requestsPerSecond` is a client-side request cap, to stay under the API’s fair-access ceiling. Omit to leave pacing to the source worker’s defaults.
`rest.verify`	Required. The reachability probe: a `path` and optional `query` the operator `GET`s to verify the API — and the credential, when `auth` is set.

rest holds access only; which endpoint, query, pagination, and field mapping to read belongs to the pipeline source — one warehouse serves many endpoints on the same API.

Verification

The operator probes the warehouse on apply, whenever the referenced Secret’s content changes, and every verifyInterval.

For snowflake, the probe opens a key-pair session, runs SELECT 1 on the declared compute warehouse, and closes.

For huggingface, the probe calls the Hub dataset API with the optional token. If pipelines reference the warehouse, the operator resolves each declared repo@revision and records the resolved commit SHA in status.sourceRevisions. With no consumers yet, it verifies that the Hub dataset API is reachable.

For rest, the probe issues a single GET of rest.verify.path (with the optional query, and auth applied when set). Any 2xx is Verified, confirming both reachability and — when a credential is configured — that it is accepted.

Phase	Meaning
`Pending`	Not yet probed.
`Verified`	Last probe succeeded; `status.verifiedAt` is the probe time.
`Failed`	Last probe failed; `status.failureReason` says why.

Failed is a loud signal, not an outage: in-flight pipeline runs keep their connections, new runs refuse to start, and the condition surfaces in kubectl get warehouse and the dashboard. Pipelines start only against a Verified warehouse.

Rotation

Swap the referenced Secret’s content. The operator re-verifies and status.verifiedAt advances; consumers resolve credentials through the warehouse at connection-build time, so new connections pick up the new key with no redeploy. Pointing keyPairSecretRef or tokenSecretRef at a different Secret name is a spec edit with the same flow.

For huggingface, swapping the token Secret’s content re-verifies the same way. Adding or removing tokenSecretRef — moving a warehouse between anonymous and authenticated access — is a spec edit the operator re-verifies.

Pipeline source

A pipeline extracting from a warehouse names it in spec.sourceRef. The source block owns the what — for Snowflake a database, query, and cursor; for Hugging Face a dataset, split, and field mapping — and the warehouse owns the where and who. The operator requires warehouseRef to name a Verified warehouse of the matching kind in the same namespace, and carries the source block verbatim to the worker as HEVLAYER_SOURCE_REF, as for any other source.

Snowflake

spec:
  sourceRef:
    kind: snowflake
    warehouseRef: prod-snowflake
    database: ANALYTICS
    query: >-
      SELECT ID, TITLE, BODY, REFRESH_ID FROM PUBLIC.NOTES
      WHERE REFRESH_ID > :cursor
    cursor:
      column: REFRESH_ID

The operator mounts the warehouse’s key-pair Secret into the worker pod at /var/run/hevlayer/warehouse/ and injects HEVLAYER_WAREHOUSE — connection JSON resolved from the warehouse spec (account, user, role, compute warehouse, pool), no credential material. The worker builds its own connection from the two.

Hugging Face

spec:
  target:
    namespace: squad
  sourceRef:
    kind: huggingface
    warehouseRef: hf-public
    dataset: rajpurkar/squad         # Hub repo id
    config: plain_text               # dataset config; omit for the default
    split: train                     # train | validation | test | …
    revision: ~                      # omit → the operator resolves and records the parquet-ref commit
    mapping:
      id: id                         # column → document id
      text: context                  # column to index or embed
      attributes: [title, question]  # columns to carry as attributes

`mapping` field	Purpose
`text`	Required. The column indexed, and embedded by a following stage.
`id`	Optional. Column used as the document id. Omitted, the worker synthesizes `{config}/{split}#{offset}` — stable within a revision; name a natural key for anything long-lived.
`attributes`	Optional. Columns carried as attributes. Omit or `[]` for every remaining scalar column; binary feature columns (image, audio) are skipped.

spec.worker.image defaults to the stock Hugging Face source image from the mesh-account ECR registry, so no custom image is needed; set it to override. The operator mounts the warehouse’s token Secret (when present) into the worker and injects HEVLAYER_WAREHOUSE with the endpoint only, no token. The worker streams rows from the dataset’s Parquet conversion at the pinned revision; the row offset is the cursor.

Omitting revision pins to the dataset’s current parquet-ref commit, which the operator resolves and records, so a long run never drifts onto a newer version mid-flight and re-enumeration is exact. Reading a newer version is a deliberate revision edit.

A dataset’s text column is often a whole document that must be split into chunks before it is embedded. Add a chunk block to the source to declare how — it is a pipeline-source feature, not specific to Hugging Face. See Chunking on the Pipeline CRD page.

spec:
  sourceRef:
    kind: huggingface
    warehouseRef: huggingface-hub
    repo: McAuley-Lab/Amazon-Reviews-2023
    config: raw_meta_Electronics
    split: train
    revision: main
    cursor:
      field: parent_asin

When sourceRef.kind is huggingface, the operator requires warehouseRef to name a Verified Hugging Face warehouse in the same namespace and requires repo on the source block. It injects HEVLAYER_WAREHOUSE with the Hub endpoint and, when tokenSecretRef is set, mounts the Secret at /var/run/hevlayer/warehouse/token and adds tokenPath to the connection JSON. HEVLAYER_SOURCE_REF remains the verbatim source block and owns the dataset repo, config, split, revision, and cursor.

REST

spec:
  target:
    namespace: drug-labels
  sourceRef:
    kind: rest
    warehouseRef: openfda
    request:
      path: /drug/label.json
      query:                           # static query params
        search: 'openfda.product_type:"HUMAN PRESCRIPTION DRUG"'
    pagination:
      kind: offset                     # offset only in v1 (searchAfter, link: not yet supported)
      pageSizeParam: limit             # required for offset
      offsetParam: skip                # required for offset
      pageSize: 1000                   # required for offset; must be > 0
    response:
      items: $.results                 # JSONPath to the record array on each page
    cursor:
      field: effective_time            # incremental + re-enumeration key (JSONPath per item)
    mapping:
      id: openfda.spl_set_id           # JSONPath into each item
      text: ~                          # omit when a chunk strategy supplies the section text
      attributes: [openfda.generic_name, openfda.brand_name, dea_schedule]

Source field	Purpose
`request.path` / `request.query`	The endpoint (resolved against `baseUrl`) and any static query params — the what.
`pagination`	How to walk pages. v1 supports `kind: offset` only — `pageSizeParam`, `offsetParam`, and `pageSize` (a positive integer) are all required. `searchAfter` (a cursor token the API echoes) and `link` (a `Link` header) are not yet supported and are rejected on apply.
`response.items`	JSONPath to the record array on each page.
`cursor.field`	JSONPath into each item, used as the incremental refresh key and the re-enumeration cursor.
`mapping`	JSONPath expressions for `id`, `text`, and `attributes` — the same mapping shape as Hugging Face, over JSON instead of Parquet columns.

spec.worker.image defaults to the stock REST source image from the mesh-account ECR registry, which pages any JSON API by these rules — there is no per-API worker. The operator injects HEVLAYER_WAREHOUSE with the baseUrl (and mounts the auth Secret when set); HEVLAYER_SOURCE_REF is the verbatim source block. A text field that is a whole document is split by a chunk block, as for any source; the section strategy with sectionSource: jsonFields makes each top-level JSON field its own section.

Keys

An ApiKey binds to a warehouse with a warehouse.<name> entitlement carrying a list of opaque claims strings. Layer stores and echoes the strings; the application routes on them. No client route reaches a source system — clients touch indexes, not warehouses — so the entitlement grants nothing in Layer, and it inerts when the warehouse is deleted.

Deletion

Deleting a warehouse fences everything drawing from it. A finalizer blocks deletion while status.consumers is non-zero — pipelines extracting from it or keys entitled to it — annotate with hevlayer.com/force-delete: "true" to override.

Status

status:
  phase: Verified
  verifiedAt: "2026-06-10T00:00:00Z"
  failureReason: null
  sourceRevisions:
    McAuley-Lab/Amazon-Reviews-2023@main: 2b6d039ed471f2ba5fd2acb718bf33b0a7e5598e
  consumers:
    pipelines: 2
    apiKeys: 1

The operator emits Kubernetes Events on phase transitions and counts observed references in status.consumers.