Enrollment troubleshooting

This page turns enrollment symptoms into diagnosis boundaries: what state to expect, which safe inspections to run, and — just as important — where to stop and escalate instead of improvising. It assumes the flow in the OIDC JWKS enrollment runbook.

Two rules apply to every symptom here:

• Inspect, don’t mutate. Diagnosis uses the Control console and non-secret CLI output only. Never print tokens, proofs, private keys, or projected token file contents while debugging.
• A missing effect is not always a failure. The lifecycle ladder (draft → validated → active → enrollment policy → enrollment → derived key → runtime propagation/readiness) has legitimate in-between states. Identify the rung you are on before acting.

Symptom index

Symptom	Likely cause	Safe response	Notes
Policy reference not found / rejected	Stale or hand-assembled opaque policy reference	Regenerate the command in Control; never reconstruct the reference	Regenerating the command resolves this
Proof file rejected as unreadable or “not a regular file”	Old CLI rejects the Kubernetes Atomic Writer symlink chain	Upgrade to CLI v0.2.1+ (v0.2.4 current), which validates the resolved descriptor	Resolved by the CLI upgrade
Rate-limited responses during enrollment or polling	Shared proxy/gateway rate-limit bucket (Envoy or similar)	Honor backoff; bounded retries only; never blind re-submission	Expect this behind shared gateways; trusted-proxy configuration is required for production deployment
Derived key exists but signing profile unavailable; no active route despite active grants	Policy/materialization failure between key, grants, and profile	Inspect state in Control; do not revoke/re-enroll as a workaround	Current limitation — requires inspection, not workarounds
Source or target stuck publishing / key active but requests denied	Runtime state not yet acknowledged by the adapter, or a stalled publication	Run the propagation checks; escalate if no terminal state is reported	Current limitation — publishing state may lack a terminal explanation
Derived-key count differs from the key list	Read-model or counting-predicate mismatch, or stale state	Compare source summary against active, non-revoked, unexpired key records; escalate the discrepancy	Current limitation
Signed request denied missing_issuer_key (403)	Passport signed under key_id (ta-key-*) instead of the resolve-issued signing kid (ta-g1-*)	Sign under the profile’s signing_kid from the workload-profile resolve response	Resolved by CLI v0.2.4+, which parses signing_kid/kid
Signed request denied replay_consume_failed (403)	Passport lifetime exceeds the adapter’s replay window	Set trustplane sign --ttl at or below the adapter replay lifetime (≤ 5m is safe)	Not a nonce reuse (jti_replay is that); the replay store rejects over-long passports

Stale or wrong opaque policy reference

Symptom. trustplane enroll fails immediately at policy lookup — the reference is reported as unknown, not found, or not valid for enrollment.

Cause. The value passed to --policy is not the current opaque Enrollment Policy reference. The policy ID visible in the Control UI is not necessarily the opaque enrollment policy_ref — the reference is generated, and it changes when the policy or its underlying source revision changes. Common ways to get here: reusing a command from an old ticket or shell history, copying an internal policy or source ID out of the UI, or hand-assembling a reference from parts.

Expected state. In Control, the enrollment policy exists, is bound to the exact active source revision, and the Enrollment Command Generator produces a complete command for it.

Safe diagnosis.

1. Open the policy in Control and confirm it is active and bound to the currently active source revision (not an older draft or superseded revision).
2. Open the Enrollment Command Generator and generate a fresh command.
3. Compare only whether your reference matches the freshly generated one — then discard the old value and run the freshly generated command verbatim.

Stop and escalate if a freshly generated reference is also rejected: that is a Control-side inconsistency between the generator and the enrollment endpoint, not an operator input error.

Projected token symlinks (Kubernetes Atomic Writer)

Symptom. The CLI refuses the proof file: unreadable, unresolvable, or rejected as not a regular file — even though ls shows the projected token mounted where you expect it.

Cause. Kubernetes maintains projected volumes through the Atomic Writer, which serves files behind a chain of symlinks (token → ..data/token → ..<timestamp>/token) so rotations are atomic. Projected token files may therefore be symlinks at the configured path. An incompatible or outdated CLI that requires the literal path to be a regular file rejects them.

Expected state. A compatible CLI (v0.2.1+; v0.2.4 is current) opens the configured path and validates the resolved file descriptor as a regular file — it accepts the projected-token symlink chain by design.

Safe diagnosis.

1. trustplane --version inside the workload — confirm v0.2.1 or later. This is the fix for nearly all cases.
2. Confirm the pod spec mounts the projected volume without subPath (updates do not propagate through subPath mounts) and that the --proof-file / --kubernetes-token-file path matches the mountPath + path in the pod spec.
3. Check file metadata only (existence, type, permissions). Do not cat or copy the token to “see if it looks right” — it is proof material.

Stop and escalate if a v0.2.1+ CLI still rejects the resolved file: capture the exact error string and the pod spec’s volume section (no token contents) and escalate.

Enrollment rate limiting behind Envoy and gateways

Symptom. Enrollment submissions or status polls return rate-limited responses, often after an initial success, and often intermittently.

Cause. When Control sits behind Envoy or another gateway, rate-limit buckets can be shared across callers on the same proxy path — your workload can be throttled by neighboring traffic, and a burst of your own polling can throttle your subsequent polls.

Expected state. The CLI already behaves correctly here: it polls with backoff at --poll-interval, honors --wait-for-activation (default true) up to --timeout, and bounds proof submissions with --submit-retries (default 2, maximum 5). Throttling during polling is a delay, not a verdict.

Safe diagnosis.

1. Distinguish which call was throttled: the challenge/submission or the status poll. A throttled status poll is not proof that enrollment failed — the enrollment may already be accepted server-side.
2. Honor the backoff. Widen --poll-interval and extend --timeout rather than re-running the command in a loop.
3. Before any re-submission, inspect the enrollment and key state in Control: if a derived key already exists for your subject, the previous attempt succeeded and no retry is needed.

Re-submission after a possibly-successful challenge is dangerous

If a challenge or proof submission may have succeeded — a timeout mid-flight, a throttled poll, a dropped connection after submit — do not blindly re-enroll. Blind retry loops both worsen the throttling and can create duplicate or conflicting enrollment state. Check Control first; use --force-reenroll only when you have confirmed the previous attempt did not produce a key you intend to keep.

Stop and escalate if rate limiting persists after honoring backoff with a single bounded client: the proxy bucket configuration (required for production deployment — per-client limits need correct trusted-proxy forwarded-address configuration) needs an infrastructure owner, not more retries.

Policy and materialization failures

Symptom. Enrollment reported success — the derived key exists, with key ID, fingerprint, and active status — but the workload cannot sign: the signing profile is unavailable, or no route is active even though the key’s grants show as active.

Cause. The rungs between “derived key issued” and “runtime-ready” involve separate state: signing-profile eligibility, route grant materialization, and bundle compilation. A compiler/profile predicate can disagree with key/grant state, leaving a key that is real but not yet usable.

Expected state. For a runtime-ready key, Control shows all of:

Derived key: active
Key-level route grant: active
Signing profile: active
Selected target: acknowledged

Safe diagnosis.

1. In Control, inspect the derived key: status, expiry, and its effective routes.
2. Inspect the signing profile for the key: does one exist, and is it active and eligible?
3. Inspect the client’s grants versus the routes that actually materialized on the key — the key should inherit exactly the client’s protected-resource grants.
4. Confirm the exact target bundle version includes the profile/route, and check its last pull/reload.

Do not revoke and re-enroll as a workaround

Revoking a key whose profile failed to materialize destroys the evidence and usually reproduces the same stall on the next key. The key is not the broken part. Inspect and escalate with the observed states instead.

Stop and escalate when the key and grants are active but the profile or route is not, and this persists across a bundle refresh: report the key ID, fingerprint, grant states, profile state, and target bundle version (all non-secret metadata).

Pending/publishing state versus adapter acknowledgment

Symptom. A source, policy release, or target sits in publishing/pending indefinitely; or everything in Control looks active but signed requests are still denied at the adapter.

Cause. Two distinct situations share this appearance:

• Normal propagation lag — Control state is ahead of the adapter. A derived key can exist before any Auth adapter has acknowledged the newest policy/bundle. This resolves on its own.
• A stalled publication — the publication/read-model state lacks a terminal explanation: neither success nor a sanitized error is reported. This does not resolve on its own.

Expected state. After propagation, Control reports target acknowledgment (not merely health), target readiness, and the expected bundle/target version with a recent pull/reload. Remember that gateway or ALB health checks passing is not adapter bundle/version readiness — a healthy pod can still be serving the previous bundle.

Safe diagnosis.

1. Check the target’s acknowledged bundle/target version against the version Control expects, and its last pull/reload time.
2. Re-run the runbook’s propagation checks: derived key active, key-level route grant active, signing profile active, selected target acknowledged.
3. If the source shows publishing, look for a sanitized error state on the source and the target version history. A publication with no terminal state after a reasonable propagation window is a stall, not a delay.
4. If the derived-key count on a source summary disagrees with the key list, compare the summary against the list filtered to active, non-revoked, unexpired keys — a mismatch is a read-model discrepancy to report, not to “fix” by deleting keys.

Stop and escalate when a publication has no terminal state after your propagation window, or when an adapter repeatedly acknowledges an older version: escalate with the source state, expected and acknowledged versions, and reload timestamps. Do not suspend/retire the source or churn enrollments to force the state machine forward.

Signed request denied missing_issuer_key

Symptom. Enrollment succeeds and the derived key is active, but the first signed request to the protected route is denied 403 {"allowed":false,"reason":"missing_issuer_key"}.

Cause. The passport was signed under the wrong key identifier. The workload-profile resolve response carries a signing kid — top-level signing_kid and, per profile, kid (format ta-g1-*) — which is the identity the adapter resolves against distributed trust material. It is distinct from key_id (ta-key-*), the Control-side handle. A passport whose header kid is set to key_id cannot be resolved by the adapter.

Expected state. The passport header kid equals the selected profile’s kid (equivalently the response signing_kid), not key_id.

Safe diagnosis.

1. Resolve the signing profile and read signing_kid (top level) and the selected profile’s kid — both are ta-g1-*.
2. Confirm the passport was signed under that value, not key_id.
3. Upgrade to CLI v0.2.4+, which parses signing_kid/kid and signs under it. Clients that do not parse these fields fall back to key_id and reproduce this symptom.

Signed request denied replay_consume_failed

Symptom. The passport resolves and its signature verifies, but the request is denied 403 {"allowed":false,"reason":"replay_consume_failed"} — on the first send, and it does not clear when retried with a fresh nonce.

Cause. This is not a nonce reuse — reuse reports jti_replay. The passport’s lifetime exceeds the adapter replay store’s maximum lifetime, so the replay record is rejected before it can be consumed.

Expected state. The passport TTL is at or below the adapter’s replay lifetime.

Safe diagnosis.

1. Check the trustplane sign --ttl value. The CLI default is long enough to exceed the adapter’s replay window and will always fail this way.
2. Re-sign with a short TTL — --ttl 5m is safe. A passport bounds a single request and should be short-lived regardless.

When any of this becomes an incident

Escalate immediately, outside this page’s scope, if you observe: a derived key you did not enroll, a key active past its expected expiry, or grants on a derived key that exceed the bound client’s grants. Those are not propagation issues — treat them under key lifecycle and compromise handling (revocation first, then diagnosis).

Next steps

• OIDC JWKS enrollment runbook — the end-to-end flow these boundaries protect.
• Key lifecycle and compromise model — expiry, rotation, revocation, and compromise response.
• Trust Anchor source availability — which enrollment profiles are supported and which are not currently supported.
• Network hardening — trusted-proxy and rate-limit prerequisites for production gateways.