Troubleshooting¶

Diagnose and resolve common issues with Bindy DNS operator.

Quick Diagnosis¶

Check Overall Health¶

# Check all resources
kubectl get all -n bindy-system

# Check CRDs
kubectl get bind9instances,dnszones,arecords -A

# Check events
kubectl get events -n bindy-system --sort-by='.lastTimestamp' | tail -20

View Status Conditions¶

# Bind9Instance status
kubectl get bind9instance primary-dns -n bindy-system -o yaml | yq '.status'

# DNSZone status
kubectl get dnszone example-com -n bindy-system -o yaml | yq '.status'

Common Issues¶

See Common Issues for frequently encountered problems and solutions.

DNS Record Label Matching Issues¶

If you're seeing "No matching DNSZone found" errors: - Records use labels to match DNSZones via label selectors - Common mistake: Record missing required labels or labels not matching DNSZone selector - See DNS Record Issues - Record Not Matching DNSZone for detailed troubleshooting

Zone Requeues Forever as "unreconciled" With an Empty INSTANCES Column¶

If the operator logs Found N unreconciled instance(s) for zone ... on every cycle while the zone serves queries correctly, and kubectl get dnszones shows an empty INSTANCES column (with Bind9Instance resources showing ZONES 0), the zone's data predates its CR: every endpoint answers "zone already exists" and versions before v0.8.0-rc.3 never recorded the instance in status.bind9Instances. Fixed in v0.8.0-rc.3; on earlier versions the loop is harmless to DNS service but re-pushes the zone configuration about once a minute. Recreating the DNSZone CR does not help; upgrading does.

Records Publish Slowly in Bulk, or Stay Served After Deletion¶

Before v0.8.0-rc.3, creating hundreds of records at once could keep many of them NotSelected or unpublished for minutes, with the operator pinned at its client-side API limit (bindy_firestoned_io_kube_api_requests_total climbing at close to BINDY_KUBE_QPS). Every record write re-read each instance's RNDC key Secret and Endpoints, and record reconciles woke each other through the zone's status.records. A deleted record could also stay in DNS when its finalizer's cleanup failed under that load, because the zone's backup cleanup gave up after one attempt.

From v0.8.0-rc.3 (ADR-0015) a write reads each instance's key and endpoints once per reconcile, a zone reconcile no longer touches every record, and a deleted record stays listed in its zone's status.records until its data is confirmed gone from every primary. If a deleted record is still answered, check the zone's events and the operator log for:

N deleted record(s) of zone <namespace>/<zone> may still be served; their DNS cleanup is retried

That warning repeats on every zone retry (the zone schedules the retry itself with capped backoff, ADR-0016) until every primary endpoint is reachable again; the record is then removed from DNS and from the status.

A Record or Zone Changed Inside BIND9 Is Not Put Back¶

From the release that implements ADR-0016 the operator has no periodic resync. It used to re-push every record every 5 minutes, which also reverted changes made directly in BIND9. Now a change made inside a running BIND9 pod (nsupdate, rndc delzone, a direct bindcar call) raises no Kubernetes event, so it stands until the next event for the resource that owns the data:

a change to the resource's spec, labels, annotations or finalizers;
the BIND9 pod being replaced (the zones-loaded gate loads every live zone and its records onto the new pod before it is Ready, ADR-0017);
the operator restarting (every object is reconciled from the initial list).

To force the repair now, change any annotation on the resource. The documented annotation is bindy.firestoned.io/reconcile-trigger; its value only has to differ from the previous one:

# One record: re-push it to every primary
kubectl annotate arecord www -n dns \
  bindy.firestoned.io/reconcile-trigger="$(date +%s)" --overwrite

# A zone: re-check it on every instance; a missing zone is re-created and all
# of its records are replayed
kubectl annotate dnszone example-com -n dns \
  bindy.firestoned.io/reconcile-trigger="$(date +%s)" --overwrite

# Every record of one kind in a namespace
kubectl annotate arecords --all -n dns \
  bindy.firestoned.io/reconcile-trigger="$(date +%s)" --overwrite

A zone that still exists inside BIND9 is not replayed by the zone annotation alone (its records are only replayed when the zone had to be re-created), so annotate the records themselves to re-push records deleted from a live zone.

Anything else that waits is woken by the object it waits on, not by a timer: a NotSelected record by a zone tagging it, a ZoneNotFound or NoPrimaryInstances record by its zone's next status change, a zone with no instances by a matching Bind9Instance, a DuplicateZone zone by the other claimant's change or deletion. A record whose write BIND9 rejected is retried with backoff, never sooner than 30 s.

A BIND9 Pod Stays Not Ready (Zones-Loaded Readiness Gate)¶

Every BIND9 pod carries the readiness gate bindy.firestoned.io/zones-loaded (ADR-0017). Kubernetes reports the pod Ready, and routes its Service to it, only once the operator has loaded every live zone of the pod's instance onto it. A pod whose containers are all running but which shows READY 2/2 with the pod still out of its Service, or kubectl rollout status waiting, is held by the gate. Read the condition:

kubectl get pod <pod> -n <namespace> -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
# or
kubectl get pod <pod> -n <namespace> -o wide   # READINESS GATES column: 0/1 or 1/1

`reason`	Meaning	What to do
(no condition)	The operator has not evaluated the pod yet. Normal for the seconds before the containers are ready.	If it lasts: is the operator running and leader? Can it patch `pods/status` (`kubectl auth can-i patch pods/status -n <namespace> --as=system:serviceaccount:bindy-system:bindy`)? An operator older than the pod template does not know the gate.
`ZonesLoading`	The operator is writing the zones and records to the pod.	Wait; large zones take as long as their record replay.
`ZonesLoadFailed`	A zone could not be loaded on this pod while another pod of the instance still serves it. The old pod keeps serving; the rollout waits. Retried with backoff (2 s to 60 s).	The message names each zone and the error. Check the zone's `DNSZone` status and the operator log (`Zones-loaded gate:`).
`InstanceUnknown`	The pod's `Bind9Instance` is not in the operator's cache.	Check that the instance exists; retried with backoff.
`ZonesLoaded` / `NoZones`	Gate open: every live zone loaded, or no live zone selects the instance.	Nothing.
`PodTerminating`	The pod is being deleted. The operator closed its gate at the start of termination (ADR-0017 decision 6). The pod's `Ready` follows on the kubelet's next status sync, so `kubectl get pods` may show it `2/2` for several seconds, then `0/2` (or `1/2`) until it is gone.	Nothing: expected for every terminating BIND9 pod.
`ZonesPartiallyLoaded`	Gate open, but the zones in the message could not be loaded and no other pod of the instance served them either.	Fix those zones; the `DNSZone` controller keeps retrying them.

The gate is evaluated once per pod: once True it stays True for the pod's life, including across container restarts, until the pod is deleted (then PodTerminating). A zone created later is configured on the running pod by the DNSZone controller as usual.

If the operator is down, new BIND9 pods stay not Ready on purpose: with the default rolling update (maxUnavailable 0) the old pod keeps serving until the operator is back.

A Bind9Instance Does Not Roll Out (`RolloutQueued`)¶

A pod-template change (a cluster configuration change, an image or bindcar change, a bindy upgrade that renders differently) is applied to one instance at a time among instances that serve a zone in common or belong to the same cluster (ADR-0018). An instance waiting its turn shows:

kubectl get bind9instance <name> -n <namespace> \
  -o jsonpath='{range .status.conditions[?(@.type=="Rollout")]}{.status} {.reason}: {.message}{"\n"}{end}'
# False RolloutQueued: Pod template change waits for Bind9Instance dns/primary-0, which is rolling out (...)

Its pods keep serving the previous configuration and its Ready condition is unaffected. It rolls as soon as the named instance finishes:

The message says the blocker...	What to check
is rolling out	Normal: `kubectl rollout status deployment/<blocker> -n <namespace>`. A blocker held by its zones-loaded gate shows why in its new pod's `bindy.firestoned.io/zones-loaded` condition (section above).
is starting its rollout	Normal, lasts until the blocker's patch reaches the operator's cache (well under a second).
was queued earlier	Instances roll first come, first served; the blocker itself waits for another one. Follow the chain through each `Rollout` condition.

A blocker whose rollout never completes stops blocking once its Deployment reports ProgressDeadlineExceeded (after progressDeadlineSeconds, 600 s by default); the waiting instance then rolls and reports Rollout=True, reason: RolloutPeerStalled. Replica changes and new instances are never queued.

An Instance Stays `RolloutQueued` After Every Rollout Finished¶

Symptoms, seen on v0.8.0-rc.6: instances keep Rollout=False/RolloutQueued naming each other although every Deployment is rolled out, the operator logs "has a pod template change queued behind Bind9Instance ..." many times a minute, and the API server sees a steady stream of Deployment PATCHes that change nothing (metadata.generation does not move).

Cause: the drift check saw a pod-template difference that the API server's defaulting erases (the bindcar container's resources: {} against none rendered), so every reconcile tried to roll and the rollout queue passed the turn around forever. Fixed by ADR-0018 decision 8: owned fields are compared semantically, and a template patch that bumps no generation is remembered and not sent again. After upgrading, an affected instance clears its Rollout condition on its next reconcile. The "queued behind" line is now logged at INFO only when an instance joins the queue (DEBUG on later wakes).

If a comparison miss ever recurs, the operator logs it once per instance and Deployment generation at WARN:

Deployment <ns>/<name>: the pod template patch for <field> changed nothing (generation <n> unchanged); treating the template as up to date

<field> names what the drift check compared. Report it with the Deployment's spec.template (kubectl get deployment <name> -n <ns> -o yaml): it is a comparison bug, and the instance keeps working meanwhile.

A Secondary Holds No Copy of the Zone (`SecondaryNotLoaded`)¶

Symptoms, seen on v0.8.0-rc.7 after pods were replaced: the secondary answers SERVFAIL for the zone; rndc zonestatus on it says the zone is not loaded; its log shows transfer of '<zone>' from <ip>#5353: failed ... denied or host unreachable; the primaries' rndc showzone <zone> names an old secondary IP in allow-transfer, and the secondary's names old primary IPs in primaries. rc.7 still reported the zone Ready=True.

Cause: the peer lists were written only when a zone was created on a pod and never updated when pods were replaced. Fixed by ADR-0019: the operator records the peers it last pushed in status.transferPeers and, when the zone's pods move, rewrites every primary's allow-transfer / also-notify and replaces every secondary zone whose primaries moved. Check:

kubectl get dnszone <name> -n <ns> -o jsonpath='{.status.transferPeers}'
# primaries:   the admitted primary pod IPs (what secondaries transfer from)
# secondaries: the secondary pod IPs (the primaries' allow-transfer)
# notify:      the secondary Services' ClusterIPs (the primaries' also-notify, port 53)

While a secondary has the zone configured but not loaded, the zone reports Degraded=True / Ready=False with reason SecondaryNotLoaded, naming the instance and endpoint, and retries with backoff until the transfer completes. TransferPeersNotUpdated means a primary refused the new allow-transfer; NoTransferSource means no primary pod has its zones loaded yet (all primaries were replaced at once and are still loading).

Zones created by an operator older than ADR-0019 carry an also-notify with a port (<ip> port 5353) that bindcar 0.9.0 cannot rewrite in place: their allow-transfer is still corrected, and the operator logs also-notify could not be once per change. Roll the BIND9 pods once after upgrading (kubectl rollout restart deployment -n <ns> -l app=bind9) to move every zone to the new form.

A Record or Zone Stays `Terminating` (`RecordDeletionPending`)¶

A record or zone deletion waits until its data is confirmed gone from every pod that holds the zone (ADR-0015 decision 7). The operator logs Pod <ip> of instance <ns>/<name> holds its zones but was not reached and the zone reports Degraded=True with reason RecordDeletionPending. This normally lasts the few seconds a restarted bindcar or named container takes to become ready again: the pod keeps its zones on its emptyDir, and skipping it used to leave the deleted record served forever (v0.8.0-rc.7). If the pod never becomes ready (a crash-looping container), fix it or delete the pod: its data goes with it and the deletion completes.

Scout Publishes Nothing for a Source, or a Scout Record Is Not Served¶

Scout's publishing limits and the bindy cluster's source binding (ADR-0025) each log one distinct message. Find it, then apply the matching fix:

# On the child (edge) cluster
kubectl logs -n bindy-system deploy/bindy-scout | grep -E 'allowed zones|allowed ranges|refused|ValidatingAdmissionPolicy'
# On the bindy cluster
kubectl logs -n bindy-system deploy/bindy | grep 'labelled for another zone'

Message	Where	Cause	Fix
`Zone is not in this Scout's allowed zones (BINDY_SCOUT_ALLOWED_ZONES), publishing nothing for this source`	Scout	The source's zone (annotation or `--default-zone`) is outside `--allowed-zones`. Records published earlier are kept.	Correct the source's `bindy.firestoned.io/zone`, or add the zone to `--allowed-zones` if this cluster really owns it. Scout picks the source up again when it changes or Scout restarts.
`Address is outside this Scout's allowed ranges (BINDY_SCOUT_ALLOWED_IP_RANGES), publishing nothing for this source` (with `ip=`)	Scout	An annotated, discovered or default address is outside `--allowed-ip-ranges`.	Correct the `bindy.firestoned.io/ip` annotation or the load balancer, or widen `--allowed-ip-ranges`. Retried every 30 seconds.
`failed to write ARecord <name>` ... `ValidatingAdmissionPolicy 'bindy-scout-source-binding' ... denied request: ServiceAccount '<sa>' may only publish into zones [...]` (or `may only write ARecords for clusters [...]`)	Scout	The bindy cluster refused the write: the account is registered for other zones or cluster names.	Register the account for this zone or name (`bindy bootstrap mc --service-account <sa> --cluster-name ... --allowed-zones ...`), or set Scout's `--allowed-zones` to match its registration so the source is refused on the edge cluster with a clearer message.
`The bindy cluster refused withdrawing from another cluster's ARecord; leaving it`	Scout	As above, for a record shared by several resources (ADR-0026): Scout tried to remove this resource's addresses from a record left under another cluster name. Not retried.	As above.
`The bindy cluster refused deleting another cluster's ARecord; leaving it`	Scout	Scout tried to clean up a record left under another cluster name, and its account is not registered for that name. Not retried.	If the record really is this cluster's leftover from a rename, add the old name to the registration (`--cluster-name new,old`) and touch the source, or delete the record by hand. Otherwise nothing to do: the record belongs to another cluster.
`... matches the selector of DNSZone <ns>/<zone> but is a Scout record labelled for another zone (...); not selecting it (ADR-0025)`	Operator	A record labelled `bindy.firestoned.io/managed-by: scout` has a `bindy.firestoned.io/zone` label different from (or missing for) the zone selecting it.	Scout never writes such a record. For a hand-made record, fix its zone label, or remove `managed-by: scout`.
`default zone ... is not in BINDY_SCOUT_ALLOWED_ZONES` / `default IP ... is outside BINDY_SCOUT_ALLOWED_IP_RANGES` / `... may not contain '*'` / `... is not a DNS name`	Scout, at startup (pod in `CrashLoopBackOff`)	The configuration contradicts itself or is malformed.	Fix the named setting; `*` is refused on purpose, leave `--allowed-zones` unset to allow every zone.
`⚠ <sa> is not registered in ConfigMap bindy-scout-sources`	`bindy bootstrap mc` output	The account was created without `--cluster-name` and `--allowed-zones`, so the admission policy does not constrain it.	Re-run with both flags.

To see what an account is registered for:

kubectl get configmap bindy-scout-sources -n bindy-system -o jsonpath='{.data}'

A Scout Name Serves Too Many, Too Few, or Flapping Addresses¶

Since ADR-0026 every resource of one cluster that publishes a DNS name adds its addresses to one ARecord. Read the record's contributor map to see who publishes what:

kubectl get arecords -n bindy-system -l bindy.firestoned.io/managed-by=scout \
  -o custom-columns='NAME:.spec.name,ADDRESSES:.spec.ipv4Addresses,CONTRIBUTORS:.metadata.annotations.bindy\.firestoned\.io/scout-contributors'
kubectl logs -n bindy-system deploy/bindy-scout | grep -E 'joins a DNS name|different TTLs|scout-contributors|kept changing'

Message or symptom	Where	Cause	Fix
`Source joins a DNS name other sources already publish; the record now holds every source's addresses`	Scout (info)	A resource resolves to a name another resource already publishes, so their addresses are merged. Expected for intentionally shared names (one Service per availability zone).	If the name should not be shared, give one resource a different `bindy.firestoned.io/record-name` or host, or opt it out.
`Sources publishing this DNS name set different TTLs; publishing the lowest`	Scout (warning)	The contributing resources set different `bindy.firestoned.io/ttl` values.	Set the same TTL on all of them, or accept the lowest.
`ARecord <name> has a malformed bindy.firestoned.io/scout-contributors annotation; fix or remove it`	Scout (reconcile error, retried)	The annotation was edited by hand. Scout will not rebuild a record from a map it cannot read.	Remove the annotation (`kubectl annotate arecord <name> -n bindy-system bindy.firestoned.io/scout-contributors-`) and touch each contributing resource so it adds itself back, or delete the record and let Scout recreate it.
`ARecord <name> kept changing while this source wrote it (5 attempts)`	Scout (reconcile error, retried)	Many resources sharing the name changed at once, or something else keeps rewriting the record.	Usually clears on the retry. If not, look for another writer of the record (`kubectl get arecord <name> -o yaml`, `managedFields`).
`Ready=False`, reason `RRsetConflict`: `ARecord <ns>/<name> of cluster '<c>' has published '<name>' in <zone>; this record writes nothing until it leaves`	Bindy cluster (record status), operator warning	Two edge clusters registered for one zone publish the same name; the other cluster's record published it first (ADR-0027). Or a cluster was renamed without registering both names.	Decide which cluster owns the name: change the `record-name` or host on the other cluster, or remove the source there. For a rename, register both names (`bindy bootstrap mc --service-account <sa> --cluster-name new,old --allowed-zones ...`); the conflict clears on the next event.
Two records for one name, both Ready	Bindy cluster	A per-resource record from before ADR-0026 is waiting to be handed over to the per-name record.	It completes once the new record is `Ready`; check its status.

Debugging Steps¶

See Debugging Guide for detailed debugging procedures.

FAQ¶

See FAQ for answers to frequently asked questions.

Getting Help¶

Check Logs¶

# Operator logs
kubectl logs -n bindy-system deployment/bindy --tail=100

# BIND9 instance logs
kubectl logs -n bindy-system -l instance=primary-dns

Describe Resources¶

# Describe Bind9Instance
kubectl describe bind9instance primary-dns -n bindy-system

# Describe pods
kubectl describe pod -n bindy-system <pod-name>

Check Resource Status¶

# Get detailed status
kubectl get bind9instance primary-dns -n bindy-system -o jsonpath='{.status}' | jq

Escalation¶

If issues persist:

Check Common Issues
Review Debugging Guide
Check FAQ
Search GitHub issues: https://github.com/firestoned/bindy/issues
Create a new issue with:
Kubernetes version
Bindy version
Resource YAMLs
Operator logs
Error messages

Next Steps¶

Common Issues - Frequently encountered problems
Debugging - Step-by-step debugging
FAQ - Frequently asked questions