Operations

ApiKey CRD

An ApiKey is a minted credential as a resource. Layer owns the credential lifecycle — mint, verify, revoke, expire — and what the key opens is declared per resource: each entitlement names a VectorStore, a Warehouse, an Agent, or Layer itself, and carries the scopes and claims for that target. Claims are opaque to Layer — an external system can use Layer as its key store and keep authorization decisions to itself.

Keys have two authoring surfaces that round-trip through one schema: kubectl get apikey -o yaml and GET /v2/keys/{keyId} are two spellings of the same object.

apiVersion: hevlayer.com/v1alpha1
kind: ApiKey
metadata:
  name: cohort-reader
  namespace: layer
spec:
  owner: acme
  description: cohort read access
  entitlements:
    vectorstore.prod-turbopuffer:
      scopes: [read]
      namespaces: ["cohort-*"]
    warehouse.prod-snowflake:
      claims:
        - "notes:cohort:*:read"
  expiresAfter: 365d
status:
  keyId: 0a1b2c3d-…
  phase: Active
  lookupHash: sha256:…
  createdAt: "2026-06-10T00:00:00Z"
  expiresAt: "2027-06-10T00:00:00Z"
  secretRef:
    name: apikey-cohort-reader

Spec

Field	Purpose
`owner`	Optional free-form owner label, echoed in list and authenticate responses.
`description`	Optional free-form description.
`entitlements`	Map keyed by target resource. Each entry carries `scopes`, `namespaces`, and `claims` for that target.
`expiresAfter`	Duration or `never`. Defaults to `365d`; `status.expiresAt` is computed at mint.

Entitlements

Key	Target
`vectorstore.<name>`	Data-plane access through the named store. `scopes` (`read`, `write`) gate routes whose `Index` resolves to that store; `namespaces` globs constrain which upstream namespaces.
`warehouse.<name>`	A list of opaque `claims` strings bound to the source system. Layer stores and echoes them; the application routes on them. No client route reaches a source — clients touch indexes, not warehouses — so the entitlement grants nothing in Layer and inerts when the warehouse is deleted.
`agent.<name>`	Invocation of the named `Agent`: the right to call `POST /v2/agents/<name>/query`. Data access is separate — the agent’s reads run under the caller’s own `vectorstore` grants — so this governs who may invoke, not what it reads. Inerts when the agent is deleted.
`layer`	The control plane itself. `scopes: [admin]` covers key management and Pipeline/Function create/delete/control routes, and satisfies `read` and `write` everywhere.

Scope meanings match inbound auth: read covers query, fetch, scans, and metrics; write covers namespace writes and worker routes.

claims is a list of opaque strings, allowed on any entitlement and the only field on a warehouse entitlement. Layer stores them, returns them from list, get, and authenticate, and never interprets them — an existing permission grammar (service:resource_type:resource_id:action strings, a legacy entitlement vocabulary) drops in verbatim, and the consuming application maps them to its own authorization.

An entitlement whose target does not exist grants nothing and surfaces as a status condition (EntitlementTargetMissing) — not an admission error, so keys and their targets can be applied in either order. Check the condition after applying: a typo in a target name looks the same as a missing target.

A key whose entitlements carry only claims — no scopes — is a pure external-store key: it authenticates, but opens no Layer route.

Minting

REST. POST /v2/keys generates the token, creates the ApiKey resource, and returns the token in the response — once. The raw token is never persisted; Layer stores only one-way hashes on the resource.

POST   /v2/keys                       # 201 { keyId, …, token } — token returned once
GET    /v2/keys                       # metadata only; ?includeRevoked
GET    /v2/keys/{keyId}
POST   /v2/keys/{keyId}/revoke        # idempotent
DELETE /v2/keys/{keyId}               # hard delete; Revoked keys only
POST   /v2/keys/authenticate          # body { token } → 200 { keyId, entitlements, … } | 401

Key-management routes require a key with the layer entitlement at admin scope. POST /v2/keys/authenticate is unauthenticated by construction — the token is the credential.

CRD. Apply an ApiKey with no credential. The operator mints the token, writes it to a Secret named in status.secretRef (key token), and moves phase from Pending to Active. The Secret is the token delivery; it is owned by the ApiKey and garbage-collected with it. Rotate by minting a replacement, deploying it, and revoking the old key.

Verification

External systems present the raw token to POST /v2/keys/authenticate and get back keyId (a stable actor id) plus the full entitlements map, then make their own authorization decisions from the claims. The gateway also accepts any Active key’s token as a bearer on its own routes, enforcing the entitlement for the store or control-plane surface the route resolves to.

Verification is one indexed lookup plus one hash check against a watch-fed in-memory map — the hot path never reads the control plane per request. status.lastSeenAt advances at most once per five minutes per key.

Phase	Meaning
`Pending`	CRD-authored key awaiting mint.
`Active`	Verifiable; token works.
`Revoked`	`POST /v2/keys/{keyId}/revoke` was called; token refused.
`Expired`	`status.expiresAt` passed; token refused.

Revoke is the default lifecycle end-state. Revoked ApiKey resources are retained indefinitely for audit and are not automatically garbage-collected. Operators should set a retention policy appropriate to their obligations. Hard delete is rare and deliberate: DELETE /v2/keys/{keyId} accepts only a Revoked key and permanently removes the resource, its lifecycle record, and any owned token Secret. Other phases return 409 Conflict; revoke first, then delete only after the retention period has passed or policy requires the record itself to be forgotten.

Deleting a VectorStore or Warehouse inerts every entitlement that names it: the keys stay Active for their other entitlements, and the deletion is finalizer-guarded on the target’s side while keys still reference it.

Backup and migration

Use a full namespace Velero backup, or an equivalent etcd snapshot, for portable ApiKey migration. Do not export and apply YAML: the verifier lives in status, and apply paths do not preserve that subresource.

Back up these Layer namespace objects:
- ApiKey resources with status: status.keyId, phase, lookupHash, and tokenHash are the credential record; the raw token is not recoverable.
- The chart’s *-keys Secret: it holds LAYER_KEY_PEPPER, which must match every restored verifier.
- Token-delivery Secrets referenced by CRD-authored keys through status.secretRef.
Restore ApiKey status explicitly. Velero restores specs by default and drops status unless the restore uses --status-include-resources apikeys.hevlayer.com, or the Restore sets spec.restoreStatus.includedResources: ["apikeys.hevlayer.com"].
Restore the namespace, including the pepper Secret, before running Helm on the target release. If Helm touches an empty namespace first, it can generate a new random pepper, making restored argon2id verifiers unusable.
If source and target Velero servers share object storage, wait until the target server lists the completed source backup before creating the restore; otherwise its local Backup object may not have synced.
Refresh source Velero discovery before backup. Install Layer CRDs before Velero discovers resources, or restart the source Velero deployment after CRD installation and wait for readiness; stale discovery can produce a Completed backup that omits ApiKey.
Quiesce the Layer operator before backup and keep it quiesced through initial restore. Velero restores ApiKey spec and status separately; an operator that starts in that window can mark a transient spec-only key MintBlocked or Pending.
Inspect the completed backup from inside the Velero server’s cluster. Its item list must include apikeys.hevlayer.com, expected ApiKey objects, the chart’s *-keys pepper Secret, and token-delivery Secrets referenced by status.secretRef.
Restart gateway/operator discovery after restore so the watch-fed key map rebuilds from restored ApiKey status and Secrets before traffic depends on the migrated tokens.
Treat MintBlocked as a failed restore, not a key rotation. The operator refuses to mint a replacement when status.keyId is empty but the deterministic delivery Secret exists; it writes a MintBlocked condition and warning Event instead. A new CRD-authored ApiKey with no delivery Secret still mints normally.

Validate the key store before accepting traffic:

SOURCE_CONTEXT=layer-a \
TARGET_CONTEXT=layer-b \
LAYER_NAMESPACE=layer \
SOURCE_BASE_URL=https://source.example.com \
TARGET_BASE_URL=https://target.example.com \
SOURCE_ADMIN_KEY=hvl_... \
scripts/apikey-velero-migration.sh

The script mints REST and CRD-authored keys, backs up the namespace, waits for the target Velero server to sync the completed backup, restores with --status-include-resources apikeys.hevlayer.com, and asserts:

original active tokens still authenticate
revoked and expired tokens remain refused
the completed backup contains expected ApiKey resources and Secrets
the source operator was quiesced and the expired control key reached Expired before backup
the restored pepper Secret matches the source
restored key status fields are present, so spec-only, incomplete, racing, or misordered restores fail loudly

Local reproduction currently needs the temporary test-license setup from the /v2/keys test kid keygen work in PR #199.

Velero backups are credential-bearing: they contain one-way verifiers, the pepper, and any remaining CRD-authored delivery Secrets. Store and encrypt them like other credential material.

Kubernetes RBAC

CRD authoring makes kubectl a minting surface, so the chart ships roles to delegate key administration without cluster-admin:

ClusterRole	Grants
`hevlayer-key-admin`	Full verbs on `apikeys`, plus `get` on delivered token Secrets. Can mint, revoke, and collect tokens.
`hevlayer-key-viewer`	`get`/`list`/`watch` on `apikeys`. No Secret access — status hashes are one-way, so viewing is audit, not credential access.

Neither role aggregates into the built-in view/edit/admin ClusterRoles: namespace viewer never silently means key viewer. Bindings are the cluster operator’s explicit act; set rbac.keyRoleBindings in Helm values to render them for the single-team case.

Bootstrapping

LAYER_GATEWAY_API_KEY is the bootstrap credential: it mints the first admin key —

spec:
  entitlements:
    layer:
      scopes: [admin]

— after which routine minting uses minted admin keys. Cluster operators can equally bootstrap by applying an ApiKey resource, since CRD authoring needs only kubectl access.