Subsystems
GitOps and policy
Turns a claim into real resources, after the org's rules have said yes.
- Crossplane v2
- ArgoCD
- Kyverno
- CEL
- Terraform
- A developer asks for a standard Postgres and never names an instance class, backup retention or network placement. Those are platform defaults, with a custom tier as the escape hatch.
- Rules are fail-closed validating policies at admission. A denied claim leaves nothing on the cluster and a POLICY_DENY row in the audit log.
- ArgoCD heals drift on anything stateless and only flags it on anything stateful, so a live database is never reverted without a human.
In the validation week
32 ms
to deny a claim over the storage ceiling. Nothing reached the cluster.
Settled by
Decision record 0006DatabaseClaim and the rule that denied one
apiVersion: platform.internal/v1alpha1kind: DatabaseClaimmetadata: name: checkout-db namespace: team-checkoutspec: parameters: tier: standard # standard | custom storageGB: 500 # the validation week's injected violation tags: team: checkout costCenter: eng-platform# gitops/policies/databaseclaim-org-rules.yaml- expression: object.spec.parameters.storageGB <= 100 message: "POLICY_DENIED rule=storage-ceiling: spec.parameters.storageGB must be at most 100"
Mechanisms
Test sheet: break the audit chain
Every row's hash covers its content and the previous row's hash. Change a field and the row fails verification. Rewrite its stored hash too, the way a careful intruder would, and the next row fails instead.
Edit any actor or outcome. Every hash is recomputed here, with the service's rule.
- prev
…stored…recomputed… - prev
…stored…recomputed… - prev
…stored…recomputed… - prev
…stored…recomputed… - prev
…stored…recomputed…
GET /v1/audit/verify
computing…
Calc sheet: score a placement
50 × cost + 30 × fragmentation + 20 × preemption, on four nodes carrying the placement contract. A latency-sensitive pod never sees the spot nodes: they are filtered before scoring runs.
kiln.platform.internal/workload-class
cpu request 1.0
- worker-aon-demand, $0.096/h, 1.2/4 cpu50 × cost1.0030 × fragmentation0.5520 × preemption1.00
86.5
- worker-bspot, $0.037/h, 0.4/4 cpu, risk 0.05
filtered before scoring: workload-class filter
- worker-con-demand, $0.170/h, 5.6/8 cpu50 × cost0.0030 × fragmentation0.8220 × preemption1.00
44.8
- worker-dspot, $0.065/h, 4.4/8 cpu, risk 0.12
filtered before scoring: workload-class filter
the SCHEDULE event this placement publishes
{"action":"SCHEDULE","details":{"outcome":"Bound","node":"worker-a","workloadClass":"latency-sensitive"}}Plan: cap the blast radius
The agent floors the percentage to whole pods from its own read of the cluster, and rejects a cap that floors to zero rather than rounding up. Faulted pods are hatched, as a section cut would be.
pods matching app=checkout-service: 7
maxReplicaPercentage: 30
floor(7 × 30%) = 2 pods faulted, deterministically, from the agent's own read of the cluster. The other 5 keep serving.
Injected failures
Oversized claim
500 GB against a 100 GB ceiling. Denied at admission; the claim never reached the cluster.
Bad deploy
A version answering 30% of requests with 500. Rolled back on the mesh, measured from the first traffic split.
SLO breach under chaos
A full network partition against a 20% error bound. Aborted, and the iptables rule was gone from the node one second later.
Invariants
Org rules are fail-closed Kyverno policies evaluated at admission, before anything is provisioned.
Decision record 0006Every resource the operator creates carries an owner reference to its custom resource, so nothing outlives it.
Decision record 0004The chaos agent re-derives the blast radius from its own cluster read, and a fault reverts on a lease dead-man switch if the controller disappears.
Decision record 0015A resilience score is SLO headroom plus recovery, reported only for a fault confirmed to have landed. A run that tested nothing never reads as a pass.
Decision record 0016Audit publishing never blocks a reconcile. A Kafka outage stalls no provisioning, rollout or revert, and a dropped publish is counted, not lost.
Decision record 0017The audit service alone computes the hash chain, under a single write credential no other subsystem holds.
Decision record 0018The operator holds create-only access to secrets.
Decision record 0019
The platform does not know, and does not need to know, whether a person or an AI agent submitted the request. An agent gets the same JSON contract, the same bearer token, the same policy gate before anything is provisioned and the same hash-chained audit row as a developer at a keyboard. That is what makes agent-submitted infrastructure safe to allow: the guardrails are in the structure of the platform, not in the judgement of the requester.
Validation
| Request type | Baseline (status quo) | p50 | p95 | n | Error rate | Guardrail |
|---|---|---|---|---|---|---|
| Provisioning (standard Postgres) | Manual Terraform PR with human review: hours to a day before merge and apply. | 11s | 16s | 10 | 0.0% (0/10) | |
| Scaling (storage growth) | Manual Terraform PR with human review: hours to a day before merge and apply. | 49 ms | 140 ms | 20 | 0.0% (0/20) | |
| Healthy deploy (canary promoted) | Manual on-call detection of a bad deploy (minutes to tens of minutes), manual rollback. | 2m14s | 2m30s | 29 | 0.0% (0/29) | |
| Bad deploy (injected regression) | Manual on-call detection of a bad deploy (minutes to tens of minutes), manual rollback. | 50s | 50s | 1 | 0.0% (0/1) | auto-rollback fired 1/1 |
| Chaos test (pod-kill, completed) | Zero pre-production resilience coverage; failure modes discovered during real incidents. | 46s | 52s | 19 | 0.0% (0/19) | |
| Chaos test (injected SLO breach) | Zero pre-production resilience coverage; failure modes discovered during real incidents. | 25s | 25s | 1 | 0.0% (0/1) | auto-abort, fault removed on the node fired 1/1 |
| Policy violation (storage over the ceiling) | Manual Terraform PR with human review: hours to a day before merge and apply. | 32 ms | 32 ms | 1 | 0.0% (0/1) | policy denial fired 1/1 |
| Audit query (actor, time range) | Manual compliance log reconstruction across disparate systems, commonly a multi-day effort at audit time. | 10 ms | 108 ms | 10 | 0.0% (0/10) |
201 audit rows in 23m13s of wall clock, 176 events acknowledged by Kafka, 0 publish failures, chain intact. Run 34085396203 on commit 78e0a2a.