Skip to content

Common Issues

Solutions to frequently encountered problems.

Bind9Instance Issues

Pods Not Starting

Symptom: Bind9Instance created but pods not running

Diagnosis:

kubectl get pods -n bindy-system -l instance=primary-dns
kubectl describe pod -n bindy-system <pod-name>

Common Causes: 1. Image pull errors - Check image name and registry access 2. Resource limits - Insufficient CPU/memory on nodes 3. RBAC issues - ServiceAccount lacks permissions

Solution:

# Check events
kubectl get events -n bindy-system

# Fix resource limits
kubectl edit bind9instance primary-dns -n bindy-system
# Increase resources.requests and resources.limits

# Verify RBAC
kubectl auth can-i create deployments \
  --as=system:serviceaccount:bindy-system:bindy

No Pods in a Namespace Other Than bindy-system

Symptom: a Bind9Instance outside the operator namespace never gets a Deployment. Its Ready condition is False with reason NotReady and a message starting Failed to create resources: ApiError: secrets is forbidden: User "system:serviceaccount:bindy-system:bindy" cannot create resource "secrets".

Cause: the operator may write Secrets only in namespaces that carry the bindy-secrets-writer Role and RoleBinding (B-5 hardening); the default install creates them in bindy-system only. The operator needs them to create the instance's <instance>-rndc-key Secret.

Diagnosis:

kubectl auth can-i create secrets -n <operand-ns> \
  --as=system:serviceaccount:bindy-system:bindy
# no = the Role/RoleBinding is missing

Solution: grant the namespace, including the matching bindcar-tokenreview binding its pods will need next. See Operands in Other Namespaces.

ConfigMap Not Created

Symptom: ConfigMap missing for Bind9Instance

Diagnosis:

kubectl get configmap -n bindy-system
kubectl logs -n bindy-system deployment/bindy | grep ConfigMap

Solution:

# Check operator logs for errors
kubectl logs -n bindy-system deployment/bindy --tail=50

# Delete and recreate instance
kubectl delete bind9instance primary-dns -n bindy-system
kubectl apply -f instance.yaml

Configuration Not Published (ConfigurationInvalid)

Symptom: a Bind9Instance or Bind9Cluster is Ready=False with reason ConfigurationInvalid and a message starting Configuration not published:. The BIND9 pods keep running.

What it means: before writing the BIND9 ConfigMap, the operator parses the named.conf files it rendered and runs a validator over them (ADR-0013). This configuration failed, so the operator did not publish it. The pods are still serving the last configuration that was published. Without the check, named would have refused to start on every pod that mounts the ConfigMap.

Diagnosis:

kubectl get bind9instance primary-dns -n bindy-system \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}{"\n"}'
kubectl logs -n bindy-system deployment/bindy | grep "not valid BIND9 configuration"

The message names the file (named.conf or named.conf.options) and the parser's error or the validator's findings (for example an allow-query entry naming an ACL that is not defined).

Solution: fix the spec field the message points at (ACLs, allowQuery, allowTransfer, listenOn, forwarders, rateLimit or dnssec). The operator retries with backoff and publishes as soon as the rendered configuration is valid. If the message points at text you did not configure, it is an operator bug: report it with the message and the resource's spec.

DNSZone Issues

Duplicate Zone Name Error

Symptom: DNSZone status shows Ready=False with reason DuplicateZone

What This Means: Multiple DNSZone custom resources are trying to manage the same DNS zone name (e.g., example.com). Bindy prevents this to avoid conflicting DNS configurations. Only one DNSZone can claim a given zone name at a time.

Diagnosis:

# Check the DNSZone status
kubectl get dnszone <zone-name> -n bindy-system -o yaml

# Look for the condition
kubectl get dnszone <zone-name> -n bindy-system -o jsonpath='{.status.conditions[?(@.type=="Ready")]}'

# Expected output showing the conflict:
# {
#   "type": "Ready",
#   "status": "False",
#   "reason": "DuplicateZone",
#   "message": "Zone 'example.com' is already claimed by other DNSZone(s): production/example-com, staging/example-com-test"
# }

Common Scenarios:

  1. Accidental Duplication: Same zone created in multiple namespaces

    # Find all DNSZones claiming the same zone name
    kubectl get dnszones --all-namespaces -o json | \
      jq -r '.items[] | select(.spec.zoneName=="example.com") | "\(.metadata.namespace)/\(.metadata.name)"'
    

  2. Migration Leftovers: Old zone not deleted after migration

    # Check for zones in all namespaces
    kubectl get dnszones --all-namespaces | grep example
    

  3. Multi-Tenant Conflicts: Different teams trying to use the same domain

    # Review zone ownership
    kubectl get dnszones --all-namespaces -o custom-columns=\
      NAMESPACE:.metadata.namespace,\
      NAME:.metadata.name,\
      ZONE:.spec.zoneName,\
      OWNER:.metadata.labels.team
    

Resolution:

  1. Identify the intended owner: Determine which DNSZone should own the zone name

    # List all conflicting zones with details
    kubectl get dnszones --all-namespaces -o json | \
      jq -r '.items[] | select(.spec.zoneName=="example.com") |
        "Namespace: \(.metadata.namespace), Name: \(.metadata.name), Created: \(.metadata.creationTimestamp)"'
    

  2. Delete duplicate zones: Remove the DNSZone resources that should NOT manage this zone

    # Delete the unwanted zone
    kubectl delete dnszone <duplicate-zone-name> -n <namespace>
    

  3. Verify resolution: Check that the intended zone is now reconciling

    # Watch the zone status update
    kubectl get dnszone <intended-zone-name> -n bindy-system -w
    
    # Expected: Status should transition from DuplicateZone to Ready
    # Ready     False   DuplicateZone   ...  (before)
    # Ready     True    ReconcileSucceeded  ...  (after)
    

Prevention:

  1. Use unique names: Name your DNSZone resources uniquely, even if managing the same domain

    # Good: Unique names even for same zone
    apiVersion: bindy.firestoned.io/v1beta1
    kind: DNSZone
    metadata:
      name: example-com-production  # ← Unique name
      namespace: bindy-system
    spec:
      zoneName: example.com  # ← Shared zone name (but only one should exist)
    

  2. Document zone ownership: Use labels to track which team/environment owns a zone

    metadata:
      name: example-com
      namespace: production
      labels:
        team: platform-engineering
        environment: production
    

  3. Implement admission control: Use a ValidatingWebhook to prevent duplicate zone creation (advanced)

Technical Details:

The duplicate zone check: - Runs at the start of every reconciliation (before any configuration changes) - Searches across all namespaces for other DNSZones with the same spec.zoneName - Excludes the current zone itself (compares by namespace/name) - Sets Ready=False with reason DuplicateZone to prevent configuration - Lists all conflicting zones in the status message for troubleshooting

This validation ensures DNS consistency and prevents: - Split-brain scenarios with different zone configurations - Conflicting DNS records from multiple sources - Accidental zone overwrites during reconciliation

No Instances Match Selector

Symptom: DNSZone status shows "No Bind9Instances matched selector"

Diagnosis:

kubectl get bind9instances -n bindy-system --show-labels
kubectl get dnszone example-com -n bindy-system -o yaml | yq '.spec.instanceSelector'

Solution:

# Verify labels on instances
kubectl label bind9instance primary-dns dns-role=primary -n bindy-system

# Or update zone selector
kubectl edit dnszone example-com -n bindy-system

Zone File Not Created

Symptom: Zone exists but no zone file in BIND9

Diagnosis:

kubectl exec -n bindy-system deployment/primary-dns -- ls -la /var/lib/bind/zones/
kubectl logs -n bindy-system deployment/bindy | grep "example-com"

Solution:

# Check if zone reconciliation succeeded
kubectl describe dnszone example-com -n bindy-system

# Trigger reconciliation by updating zone
kubectl annotate dnszone example-com reconcile=true -n bindy-system

Zone Answers NXDOMAIN (or a Public Address) After a Pod Restart

Symptom: A BIND9 pod was replaced — an operator upgrade, a placement change that rolled the Deployment, an eviction, a node reboot, a kubectl delete pod — and the server now answers for the zone but has no data in it.

Two shapes, depending on the zone's global settings:

# Without recursion: authoritative NXDOMAIN. Note the `aa` flag - the server is
# authoritative and confidently wrong, so clients do NOT fail over.
$ dig @<pod-ip> www.example.com
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 3132
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
# With `global.recursion: true` and `global.forwarders`: the query is forwarded
# upstream and the PUBLIC answer is returned for an internal name. Nothing errors,
# monitoring that only checks "does it resolve" passes, and traffic silently leaves
# the LAN.
$ dig @<pod-ip> internal.example.com
internal.example.com.  IN  CNAME  public-endpoint.example.net.

What This Means: BIND9 operand pods keep zone data in ephemeral storage, so a replaced pod comes back with no zones at all. The operator recreates the zone from spec, which yields the SOA and NS records only — roughly a 250-byte zone file. Until the zone's records are pushed back in, the server is authoritative for a zone with no data.

Diagnosis:

# The zone reports the outstanding replay
kubectl get dnszone <zone-name> -n bindy-system \
  -o jsonpath='{.status.recordsResyncPending}{"\n"}'

# ... and stays out of Ready while it does
kubectl get dnszone <zone-name> -n bindy-system \
  -o jsonpath='{.status.conditions[?(@.type=="Degraded")]}{"\n"}'
# {"type":"Degraded","status":"True","reason":"RecordsResyncPending",
#  "message":"Replayed 8/10 record(s) into zone example.com; 2 failed: ..."}

# Operator log lines for the same event
kubectl logs -n bindy-system deploy/bindy | grep -E 'was MISSING on endpoint|Record resync'

Resolution:

Bindy repairs this on its own. When the zone reconciler creates a zone (rather than finding it already present) it replays every record CR the zone selects into BIND9 in the same reconciliation, and the DNSZone controller watches Endpoints, so a pod becoming ready triggers that reconciliation within seconds.

If status.recordsResyncPending stays true across several reconciliations, the replay is failing rather than pending — read the Degraded message, which names each record that could not be pushed. The usual causes are the ones in DNS Record Issues: an unreachable pod endpoint, or an RNDC/TSIG key mismatch rejecting the dynamic update.

Why readiness matters here

A zone with an outstanding replay reports Ready=False deliberately. Clients fail over on timeouts, not on NXDOMAIN, so a server that answers confidently and wrongly is worse than one that is unreachable — nothing routes around it. Keeping the DNSZone out of Ready is what makes the condition visible to operators and to any automation gating on it.

DNS Record Issues

Record Not Matching DNSZone (Event-Driven Architecture)

Symptom: Record created but status.zoneRef is not set, or record status shows "NotSelected"

Diagnosis:

# Check if record has been selected by a zone
kubectl get arecord www-example -n bindy-system -o jsonpath='{.status.zoneRef}'

# Check record status conditions
kubectl get arecord www-example -n bindy-system -o jsonpath='{.status.conditions[?(@.type=="Ready")]}'

# Check the record's labels
kubectl get arecord www-example -n bindy-system -o jsonpath='{.metadata.labels}'

# Check available DNSZones and their selectors
kubectl get dnszones -n bindy-system

# Check the DNSZone's label selector
kubectl get dnszone example-com -n bindy-system -o jsonpath='{.spec.recordsFrom[*].selector}'

Understanding the Problem:

With the event-driven architecture, DNS records are matched to DNSZones via watch events:

  1. DNSZone watches all 9 record types (ARecord, AAAARecord, TXTRecord, CNAMERecord, MXRecord, NSRecord, SRVRecord, CAARecord, PTRRecord)
  2. When a record is created/updated, DNSZone receives a watch event immediately (⚡ sub-second)
  3. DNSZone evaluates if record labels match spec.recordsFrom selectors
  4. If matched, DNSZone sets record.status.zoneRef with full zone metadata
  5. Record operator watches for status changes and reconciles when status.zoneRef is set

Common Mistakes: - Record has label zone: internal-local but DNSZone expects zone: internal.local - Record missing the required label entirely - DNSZone spec.recordsFrom selector doesn't match any records - Typo in label key or value - Record and DNSZone in different namespaces (watches are namespace-scoped)

Expected Behavior (Event-Driven):

Record created at 10:00:00.000
  → DNSZone watch triggered at 10:00:00.050 ⚡ (immediate)
  → Label selectors evaluated
  → status.zoneRef set at 10:00:00.100 (if matched)
  → Record watch triggered at 10:00:00.150 ⚡ (immediate)
  → Record reconciles to BIND9 at 10:00:00.500
Total time: ~500ms ✅

Troubleshooting:

  1. Verify record labels match zone selector:

    # Get record labels
    kubectl get arecord www-example -o jsonpath='{.metadata.labels}'
    
    # Get zone selector
    kubectl get dnszone example-com -o jsonpath='{.spec.recordsFrom[0].selector}'
    

  2. Check if record is in the same namespace as the zone:

    kubectl get arecord www-example -n bindy-system
    kubectl get dnszone example-com -n bindy-system
    

  3. Verify DNSZone operator is running:

    kubectl logs -n bindy-system deployment/bindy | grep "DNSZone watch"
    

  4. Check record status.zoneRef field:

    # If empty/null, record hasn't been selected
    kubectl get arecord www-example -o yaml | grep -A5 "zoneRef:"
    

Solution:

Ensure record labels match the DNSZone's selector

Example:

Given this DNSZone:

apiVersion: bindy.firestoned.io/v1beta1
kind: DNSZone
metadata:
  name: example-com
  namespace: bindy-system
spec:
  zoneName: example.com
  recordSelector:
    matchLabels:
      zone: example.com  # ← Selector expects this label

Wrong:

# Record without matching label
apiVersion: bindy.firestoned.io/v1beta1
kind: ARecord
metadata:
  name: www-example
  namespace: bindy-system
  # ✗ Missing labels!
spec:
  name: www
  ipv4Addresses:
    - "192.0.2.1"

Correct:

# Record with matching label
apiVersion: bindy.firestoned.io/v1beta1
kind: ARecord
metadata:
  name: www-example
  namespace: bindy-system
  labels:
    zone: example.com  # ✓ Matches DNSZone selector
spec:
  name: www
  ipv4Addresses:
    - "192.0.2.1"

Verification:

# After fixing, check the record reconciles
kubectl describe arecord www-example -n bindy-system

# Check which DNSZone the record matched
kubectl get arecord www-example -n bindy-system -o yaml | yq '.status.zone'

# Should see no errors in events
kubectl get events -n bindy-system --sort-by='.lastTimestamp' | tail -10

See the Label Selectors Guide for more details.

Record Not Appearing in Zone

Symptom: ARecord created but not in zone file

Diagnosis:

# Check record status
kubectl get arecord www-example -n bindy-system -o yaml

# Check zone file
kubectl exec -n bindy-system deployment/primary-dns -- cat /var/lib/bind/zones/example.com.zone

Solution:

# Verify record has the correct labels
kubectl get arecord www-example -n bindy-system -o yaml | yq '.metadata.labels'

# Check DNSZone selector
kubectl get dnszone example-com -n bindy-system -o yaml | yq '.spec.recordSelector'

# Update labels to match selector
kubectl label arecord www-example zone=example.com -n bindy-system --overwrite

DNS Query Not Resolving

Symptom: dig/nslookup fails to resolve

Diagnosis:

# Get DNS service IP
SERVICE_IP=$(kubectl get svc primary-dns -n bindy-system -o jsonpath='{.spec.clusterIP}')

# Test query
dig @$SERVICE_IP www.example.com

# Check BIND9 logs
kubectl logs -n bindy-system -l instance=primary-dns | tail -20

Solutions:

  1. Record doesn't exist:

    kubectl get arecords -n bindy-system
    kubectl apply -f record.yaml
    

  2. Zone not loaded:

    kubectl logs -n bindy-system -l instance=primary-dns | grep "loaded serial"
    

  3. Network policy blocking:

    kubectl get networkpolicies -n bindy-system
    

Zone Transfer Issues

Secondary Not Receiving Transfers

Symptom: Secondary instance not getting zone updates

Diagnosis:

# Check secondary logs
kubectl logs -n bindy-system -l dns-role=secondary | grep transfer

# Check if zone has secondary IPs configured
kubectl get dnszone example-com -n bindy-system -o jsonpath='{.status.secondaryIps}'

# Check if secondaries are discovered
kubectl get bind9instance -n bindy-system -l role=secondary -o jsonpath='{.items[*].status.podIP}'

Automatic Configuration:

As of v0.1.0, Bindy automatically discovers secondary IPs and configures zone transfers: - Secondary pods are discovered via Kubernetes API using label selectors (role=secondary) - Primary zones are configured with also-notify and allow-transfer directives - Secondary IPs are stored in DNSZone.status.secondaryIps for tracking - When secondary pods restart/reschedule and get new IPs, zones are automatically updated

Manual Verification:

# Check if zone has secondary IPs in status
kubectl get dnszone example-com -n bindy-system -o yaml | yq '.status.secondaryIps'

# Expected output: List of secondary pod IPs
# - 10.244.1.5
# - 10.244.2.8

# Verify zone configuration on primary
kubectl exec -n bindy-system deployment/primary-dns -- \
  curl -s localhost:8080/api/zones/example.com | jq '.alsoNotify, .allowTransfer'

If Automatic Configuration Fails:

  1. Verify secondary instances are labeled correctly:

    kubectl get bind9instance -n bindy-system -o yaml | yq '.items[].metadata.labels'
    
    # Expected labels for secondaries:
    # role: secondary
    # cluster: <cluster-name>
    

  2. Check DNSZone reconciler logs:

    kubectl logs -n bindy-system deployment/bindy | grep "secondary"
    

  3. Verify network connectivity:

    # Test AXFR from secondary to primary
    kubectl exec -n bindy-system deployment/secondary-dns -- \
      dig @primary-dns-service AXFR example.com
    

Recovery After Secondary Pod Restart:

When secondary pods are rescheduled and get new IPs: 1. Detection: Reconciler automatically detects IP change within 5-10 minutes (next reconciliation) 2. Update: Zones are deleted and recreated with new secondary IPs 3. Transfer: Zone transfers resume automatically with new IPs

Manual Trigger (if needed):

# Force reconciliation by updating zone annotation
kubectl annotate dnszone example-com -n bindy-system \
  reconcile.bindy.firestoned.io/trigger="$(date +%s)" --overwrite

Performance Issues

High Query Latency

Symptom: DNS queries taking too long

Diagnosis:

# Test query time
time dig @$SERVICE_IP example.com

# Check resource usage
kubectl top pods -n bindy-system -l instance=primary-dns

Solutions:

  1. Increase resources:

    spec:
      resources:
        limits:
          cpu: "1000m"
          memory: "1Gi"
    

  2. Add more replicas:

    spec:
      replicas: 3
    

  3. Enable caching (if appropriate for your use case)

Operator logs "Kubernetes API request timed out"

Symptom: The operator logs Retryable Kubernetes API error, will retry with Kubernetes API request timed out after 30s (client-side request deadline), and bindy_firestoned_io_kube_api_requests_total{status="error"} rises.

Cause: A non-watch request got no complete response within BINDY_KUBE_REQUEST_TIMEOUT_SECS (default 30 s). Usually one stalled connection between the operator and the API server; the deadline drops it and the request is retried with backoff on a fresh connection (ADR-0014). Before this deadline existed, such a request hung for about five minutes.

What to do:

  1. Occasional timeouts that succeed on retry need no action.
  2. If they persist, check the network path to the API server (node network, load balancer in front of the API server, kubectl get --raw /readyz).
  3. If the API server is healthy but slow admission webhooks on bindy's resources make legitimate writes take longer than the deadline, raise BINDY_KUBE_REQUEST_TIMEOUT_SECS on the operator Deployment.

Watch streams are never cut by this deadline.

RBAC Issues

Forbidden Errors in Logs

Symptom: Operator logs show "Forbidden" errors

Diagnosis:

kubectl logs -n bindy-system deployment/bindy | grep Forbidden

# Check permissions
kubectl auth can-i create deployments \
  --as=system:serviceaccount:bindy-system:bindy \
  -n bindy-system

Solution:

# Reapply RBAC
kubectl apply -f deploy/rbac/

# Verify ClusterRoleBinding
kubectl get clusterrolebinding bindy-rolebinding -o yaml

# Restart operator
kubectl rollout restart deployment/bindy -n bindy-system

bindcar Sidecar Authentication (Mode B / TokenReview)

Sidecar crashloops: "refusing to start ... without real authentication"

Symptom: the operand pod's api container crashloops; its logs show: refusing to start: the API is bound to a non-loopback interface (0.0.0.0) without real authentication.

Cause: the bindcar image lacks the TokenReview auth path. bindy runs "Mode B" (no BIND_API_TOKEN), so the image must be built with --features k8s-token-review. Published images >= v0.7.2 include it; v0.7.0/v0.7.1 do not.

Solution: use ghcr.io/firestoned/bindcar:v0.9.0 (the current default) or any >= v0.7.2. If you pin bindcarConfig.image, bump it to v0.7.2+.

Operator gets HTTP 401 from the bindcar API

Symptom: operator logs show persistent 401 Unauthorized from bindcar zone operations (a 401 is not retried), and zones never become Ready.

Diagnosis:

# Is the operand SA allowed to create tokenreviews?
kubectl auth can-i create tokenreviews.authentication.k8s.io \
  --as=system:serviceaccount:bindy-system:bind9

# Does the operator carry the bindcar-audience projected token?
kubectl get deploy/bindy -n bindy-system -o jsonpath='{.spec.template.spec.volumes[?(@.name=="bindcar-token")].projected.sources[0].serviceAccountToken.audience}'
# expect: bindcar

Common causes & fixes: - Missing tokenreview RBAC → kubectl apply -f deploy/operator/rbac/tokenreview-clusterrole.yaml -f deploy/operator/rbac/tokenreview-clusterrolebinding.yaml (add a subject per operand namespace). - Operand outside bindy-system → the shipped binding names only system:serviceaccount:bindy-system:bind9. Re-run the can-i check with --as=system:serviceaccount:<operand-ns>:bind9, and bind that namespace's SA as described in Operands in Other Namespaces. - Audience mismatch → the operator's projected token audience must equal the sidecar's BIND_TOKEN_AUDIENCES (bindcar). - Wrong allow-list → BIND_ALLOWED_SERVICE_ACCOUNTS must name the operator SA (system:serviceaccount:<ns>:bindy), not the operand bind9 SA.

See the bindcar 0.7.x migration guide and RBAC.

Pod Restarts and Node Drains

Queries fail right after a BIND9 Pod restarts

BIND9 keeps its zone data inside the Pod (/etc/bind/zones and /var/cache/bind are emptyDir unless you attach a PersistentVolumeClaim). A replacement Pod therefore starts empty, and the operator has to push every zone and record back into it.

The readiness probe is a TCP connect to the DNS port, so it passes as soon as named accepts connections — before any zone is loaded. For a short window the Pod is in the Service endpoints and answers REFUSED.

How large that window is depends entirely on whether other primaries are still serving:

Situation Time to serve after the Pod is Ready
One primary replaced, peers still serving ~1 second
Every primary of a cluster replaced at once ~115 seconds

Keep restarts in the first row. Bindy creates a PodDisruptionBudget per cluster and role (<cluster>-primary-pdb, <cluster>-secondary-pdb) with maxUnavailable: 1, so a node drain or cluster upgrade can only take one operand at a time:

kubectl get poddisruptionbudget -n bindy-system

If a drain appears to hang, that budget is doing its job — it is waiting for the previous operand to come back before releasing the next.

The budgets only govern voluntary disruption. kubectl delete pod -l app=bind9 bypasses them entirely and will produce the slow case, as will losing several nodes at once.

Making restarts cheaper

  • Run more than one primary (spec.primary.replicas: 2 or more) so a restart never takes the last server for a zone.
  • Attach a PersistentVolumeClaim for the zone directory (see examples/bind9-cluster-with-storage.yaml) so zone data survives the Pod and does not need to be re-pushed.

Graceful shutdown

Terminating operand Pods run a preStop hook that waits before named receives SIGTERM, because Kubernetes removes the Pod from the Service endpoints and signals the container at the same time. Without that pause named can exit while kube-proxy is still forwarding queries to it. The hook then runs rndc sync -clean to flush journals, which matters when the zone directory is a PVC. terminationGracePeriodSeconds is set to comfortably outlast the drain.

Next Steps