Common Issues

Solutions to frequently encountered problems.

Bind9Instance Issues

Pods Not Starting

Symptom: Bind9Instance created but pods not running

Diagnosis:

kubectl get pods -n bindy-system -l instance=primary-dns
kubectl describe pod -n bindy-system <pod-name>

Common Causes: 1. Image pull errors - Check image name and registry access 2. Resource limits - Insufficient CPU/memory on nodes 3. RBAC issues - ServiceAccount lacks permissions

Solution:

# Check events
kubectl get events -n bindy-system

# Fix resource limits
kubectl edit bind9instance primary-dns -n bindy-system
# Increase resources.requests and resources.limits

# Verify RBAC
kubectl auth can-i create deployments \
  --as=system:serviceaccount:bindy-system:bindy

ConfigMap Not Created

Symptom: ConfigMap missing for Bind9Instance

Diagnosis:

kubectl get configmap -n bindy-system
kubectl logs -n bindy-system deployment/bindy | grep ConfigMap

Solution:

# Check operator logs for errors
kubectl logs -n bindy-system deployment/bindy --tail=50

# Delete and recreate instance
kubectl delete bind9instance primary-dns -n bindy-system
kubectl apply -f instance.yaml

DNSZone Issues

Duplicate Zone Name Error

Symptom: DNSZone status shows Ready=False with reason DuplicateZone

What This Means: Multiple DNSZone custom resources are trying to manage the same DNS zone name (e.g., example.com). Bindy prevents this to avoid conflicting DNS configurations. Only one DNSZone can claim a given zone name at a time.

Diagnosis:

# Check the DNSZone status
kubectl get dnszone <zone-name> -n bindy-system -o yaml

# Look for the condition
kubectl get dnszone <zone-name> -n bindy-system -o jsonpath='{.status.conditions[?(@.type=="Ready")]}'

# Expected output showing the conflict:
# {
#   "type": "Ready",
#   "status": "False",
#   "reason": "DuplicateZone",
#   "message": "Zone 'example.com' is already claimed by other DNSZone(s): production/example-com, staging/example-com-test"
# }

Common Scenarios:

1. Accidental Duplication: Same zone created in multiple namespaces

   # Find all DNSZones claiming the same zone name
   kubectl get dnszones --all-namespaces -o json | \
     jq -r '.items[] | select(.spec.zoneName=="example.com") | "\(.metadata.namespace)/\(.metadata.name)"'
   

2. Migration Leftovers: Old zone not deleted after migration

   # Check for zones in all namespaces
   kubectl get dnszones --all-namespaces | grep example
   

3. Multi-Tenant Conflicts: Different teams trying to use the same domain

   # Review zone ownership
   kubectl get dnszones --all-namespaces -o custom-columns=\
     NAMESPACE:.metadata.namespace,\
     NAME:.metadata.name,\
     ZONE:.spec.zoneName,\
     OWNER:.metadata.labels.team
   

Resolution:

1. Identify the intended owner: Determine which DNSZone should own the zone name

   # List all conflicting zones with details
   kubectl get dnszones --all-namespaces -o json | \
     jq -r '.items[] | select(.spec.zoneName=="example.com") |
       "Namespace: \(.metadata.namespace), Name: \(.metadata.name), Created: \(.metadata.creationTimestamp)"'
   

2. Delete duplicate zones: Remove the DNSZone resources that should NOT manage this zone

   # Delete the unwanted zone
   kubectl delete dnszone <duplicate-zone-name> -n <namespace>
   

3. Verify resolution: Check that the intended zone is now reconciling

   # Watch the zone status update
   kubectl get dnszone <intended-zone-name> -n bindy-system -w
   
   # Expected: Status should transition from DuplicateZone to Ready
   # Ready     False   DuplicateZone   ...  (before)
   # Ready     True    ReconcileSucceeded  ...  (after)
   

Prevention:

1. Use unique names: Name your DNSZone resources uniquely, even if managing the same domain

   # Good: Unique names even for same zone
   apiVersion: bindy.firestoned.io/v1beta1
   kind: DNSZone
   metadata:
     name: example-com-production  # ← Unique name
     namespace: bindy-system
   spec:
     zoneName: example.com  # ← Shared zone name (but only one should exist)
   

2. Document zone ownership: Use labels to track which team/environment owns a zone

   metadata:
     name: example-com
     namespace: production
     labels:
       team: platform-engineering
       environment: production
   

3. Implement admission control: Use a ValidatingWebhook to prevent duplicate zone creation (advanced)

Technical Details:

The duplicate zone check: - Runs at the start of every reconciliation (before any configuration changes) - Searches across all namespaces for other DNSZones with the same spec.zoneName - Excludes the current zone itself (compares by namespace/name) - Sets Ready=False with reason DuplicateZone to prevent configuration - Lists all conflicting zones in the status message for troubleshooting

This validation ensures DNS consistency and prevents: - Split-brain scenarios with different zone configurations - Conflicting DNS records from multiple sources - Accidental zone overwrites during reconciliation

No Instances Match Selector

Symptom: DNSZone status shows "No Bind9Instances matched selector"

Diagnosis:

kubectl get bind9instances -n bindy-system --show-labels
kubectl get dnszone example-com -n bindy-system -o yaml | yq '.spec.instanceSelector'

Solution:

# Verify labels on instances
kubectl label bind9instance primary-dns dns-role=primary -n bindy-system

# Or update zone selector
kubectl edit dnszone example-com -n bindy-system

Zone File Not Created

Symptom: Zone exists but no zone file in BIND9

Diagnosis:

kubectl exec -n bindy-system deployment/primary-dns -- ls -la /var/lib/bind/zones/
kubectl logs -n bindy-system deployment/bindy | grep "example-com"

Solution:

# Check if zone reconciliation succeeded
kubectl describe dnszone example-com -n bindy-system

# Trigger reconciliation by updating zone
kubectl annotate dnszone example-com reconcile=true -n bindy-system

Zone Answers NXDOMAIN (or a Public Address) After a Pod Restart

Symptom: A BIND9 pod was replaced — an operator upgrade, a placement change that rolled the Deployment, an eviction, a node reboot, a kubectl delete pod — and the server now answers for the zone but has no data in it.

Two shapes, depending on the zone's global settings:

# Without recursion: authoritative NXDOMAIN. Note the `aa` flag - the server is
# authoritative and confidently wrong, so clients do NOT fail over.
$ dig @<pod-ip> www.example.com
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 3132
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

# With `global.recursion: true` and `global.forwarders`: the query is forwarded
# upstream and the PUBLIC answer is returned for an internal name. Nothing errors,
# monitoring that only checks "does it resolve" passes, and traffic silently leaves
# the LAN.
$ dig @<pod-ip> internal.example.com
internal.example.com.  IN  CNAME  public-endpoint.example.net.

What This Means: BIND9 operand pods keep zone data in ephemeral storage, so a replaced pod comes back with no zones at all. The operator recreates the zone from spec, which yields the SOA and NS records only — roughly a 250-byte zone file. Until the zone's records are pushed back in, the server is authoritative for a zone with no data.

Diagnosis:

# The zone reports the outstanding replay
kubectl get dnszone <zone-name> -n bindy-system \
  -o jsonpath='{.status.recordsResyncPending}{"\n"}'

# ... and stays out of Ready while it does
kubectl get dnszone <zone-name> -n bindy-system \
  -o jsonpath='{.status.conditions[?(@.type=="Degraded")]}{"\n"}'
# {"type":"Degraded","status":"True","reason":"RecordsResyncPending",
#  "message":"Replayed 8/10 record(s) into zone example.com; 2 failed: ..."}

# Operator log lines for the same event
kubectl logs -n bindy-system deploy/bindy | grep -E 'was MISSING on endpoint|Record resync'

Resolution:

Bindy repairs this on its own. When the zone reconciler creates a zone (rather than finding it already present) it replays every record CR the zone selects into BIND9 in the same reconciliation, and the DNSZone controller watches Endpoints, so a pod becoming ready triggers that reconciliation within seconds.

If status.recordsResyncPending stays true across several reconciliations, the replay is failing rather than pending — read the Degraded message, which names each record that could not be pushed. The usual causes are the ones in DNS Record Issues: an unreachable pod endpoint, or an RNDC/TSIG key mismatch rejecting the dynamic update.

Why readiness matters here

A zone with an outstanding replay reports Ready=False deliberately. Clients fail over on timeouts, not on NXDOMAIN, so a server that answers confidently and wrongly is worse than one that is unreachable — nothing routes around it. Keeping the DNSZone out of Ready is what makes the condition visible to operators and to any automation gating on it.

DNS Record Issues

Record Not Matching DNSZone (Event-Driven Architecture)

Symptom: Record created but status.zoneRef is not set, or record status shows "NotSelected"

Diagnosis:

# Check if record has been selected by a zone
kubectl get arecord www-example -n bindy-system -o jsonpath='{.status.zoneRef}'

# Check record status conditions
kubectl get arecord www-example -n bindy-system -o jsonpath='{.status.conditions[?(@.type=="Ready")]}'

# Check the record's labels
kubectl get arecord www-example -n bindy-system -o jsonpath='{.metadata.labels}'

# Check available DNSZones and their selectors
kubectl get dnszones -n bindy-system

# Check the DNSZone's label selector
kubectl get dnszone example-com -n bindy-system -o jsonpath='{.spec.recordsFrom[*].selector}'

Understanding the Problem:

With the event-driven architecture, DNS records are matched to DNSZones via watch events:

1. DNSZone watches all 9 record types (ARecord, AAAARecord, TXTRecord, CNAMERecord, MXRecord, NSRecord, SRVRecord, CAARecord, PTRRecord)
2. When a record is created/updated, DNSZone receives a watch event immediately (⚡ sub-second)
3. DNSZone evaluates if record labels match spec.recordsFrom selectors
4. If matched, DNSZone sets record.status.zoneRef with full zone metadata
5. Record operator watches for status changes and reconciles when status.zoneRef is set

Common Mistakes: - Record has label zone: internal-local but DNSZone expects zone: internal.local - Record missing the required label entirely - DNSZone spec.recordsFrom selector doesn't match any records - Typo in label key or value - Record and DNSZone in different namespaces (watches are namespace-scoped)

Expected Behavior (Event-Driven):

Record created at 10:00:00.000
  → DNSZone watch triggered at 10:00:00.050 ⚡ (immediate)
  → Label selectors evaluated
  → status.zoneRef set at 10:00:00.100 (if matched)
  → Record watch triggered at 10:00:00.150 ⚡ (immediate)
  → Record reconciles to BIND9 at 10:00:00.500
Total time: ~500ms ✅

Troubleshooting:

1. Verify record labels match zone selector:

   # Get record labels
   kubectl get arecord www-example -o jsonpath='{.metadata.labels}'
   
   # Get zone selector
   kubectl get dnszone example-com -o jsonpath='{.spec.recordsFrom[0].selector}'
   

2. Check if record is in the same namespace as the zone:

   kubectl get arecord www-example -n bindy-system
   kubectl get dnszone example-com -n bindy-system
   

3. Verify DNSZone operator is running:

   kubectl logs -n bindy-system deployment/bindy | grep "DNSZone watch"
   

4. Check record status.zoneRef field:

   # If empty/null, record hasn't been selected
   kubectl get arecord www-example -o yaml | grep -A5 "zoneRef:"
   

Solution:

Ensure record labels match the DNSZone's selector

Example:

Given this DNSZone:

apiVersion: bindy.firestoned.io/v1beta1
kind: DNSZone
metadata:
  name: example-com
  namespace: bindy-system
spec:
  zoneName: example.com
  recordSelector:
    matchLabels:
      zone: example.com  # ← Selector expects this label

Wrong:

# Record without matching label
apiVersion: bindy.firestoned.io/v1beta1
kind: ARecord
metadata:
  name: www-example
  namespace: bindy-system
  # ✗ Missing labels!
spec:
  name: www
  ipv4Addresses:
    - "192.0.2.1"

Correct:

# Record with matching label
apiVersion: bindy.firestoned.io/v1beta1
kind: ARecord
metadata:
  name: www-example
  namespace: bindy-system
  labels:
    zone: example.com  # ✓ Matches DNSZone selector
spec:
  name: www
  ipv4Addresses:
    - "192.0.2.1"

Verification:

# After fixing, check the record reconciles
kubectl describe arecord www-example -n bindy-system

# Check which DNSZone the record matched
kubectl get arecord www-example -n bindy-system -o yaml | yq '.status.zone'

# Should see no errors in events
kubectl get events -n bindy-system --sort-by='.lastTimestamp' | tail -10

See the Label Selectors Guide for more details.

Record Not Appearing in Zone

Symptom: ARecord created but not in zone file

Diagnosis:

# Check record status
kubectl get arecord www-example -n bindy-system -o yaml

# Check zone file
kubectl exec -n bindy-system deployment/primary-dns -- cat /var/lib/bind/zones/example.com.zone

Solution:

# Verify record has the correct labels
kubectl get arecord www-example -n bindy-system -o yaml | yq '.metadata.labels'

# Check DNSZone selector
kubectl get dnszone example-com -n bindy-system -o yaml | yq '.spec.recordSelector'

# Update labels to match selector
kubectl label arecord www-example zone=example.com -n bindy-system --overwrite

DNS Query Not Resolving

Symptom: dig/nslookup fails to resolve

Diagnosis:

# Get DNS service IP
SERVICE_IP=$(kubectl get svc primary-dns -n bindy-system -o jsonpath='{.spec.clusterIP}')

# Test query
dig @$SERVICE_IP www.example.com

# Check BIND9 logs
kubectl logs -n bindy-system -l instance=primary-dns | tail -20

Solutions:

1. Record doesn't exist:

   kubectl get arecords -n bindy-system
   kubectl apply -f record.yaml
   

2. Zone not loaded:

   kubectl logs -n bindy-system -l instance=primary-dns | grep "loaded serial"
   

3. Network policy blocking:

   kubectl get networkpolicies -n bindy-system
   

Zone Transfer Issues

Secondary Not Receiving Transfers

Symptom: Secondary instance not getting zone updates

Diagnosis:

# Check secondary logs
kubectl logs -n bindy-system -l dns-role=secondary | grep transfer

# Check if zone has secondary IPs configured
kubectl get dnszone example-com -n bindy-system -o jsonpath='{.status.secondaryIps}'

# Check if secondaries are discovered
kubectl get bind9instance -n bindy-system -l role=secondary -o jsonpath='{.items[*].status.podIP}'

Automatic Configuration:

As of v0.1.0, Bindy automatically discovers secondary IPs and configures zone transfers: - Secondary pods are discovered via Kubernetes API using label selectors (role=secondary) - Primary zones are configured with also-notify and allow-transfer directives - Secondary IPs are stored in DNSZone.status.secondaryIps for tracking - When secondary pods restart/reschedule and get new IPs, zones are automatically updated

Manual Verification:

# Check if zone has secondary IPs in status
kubectl get dnszone example-com -n bindy-system -o yaml | yq '.status.secondaryIps'

# Expected output: List of secondary pod IPs
# - 10.244.1.5
# - 10.244.2.8

# Verify zone configuration on primary
kubectl exec -n bindy-system deployment/primary-dns -- \
  curl -s localhost:8080/api/zones/example.com | jq '.alsoNotify, .allowTransfer'

If Automatic Configuration Fails:

1. Verify secondary instances are labeled correctly:

   kubectl get bind9instance -n bindy-system -o yaml | yq '.items[].metadata.labels'
   
   # Expected labels for secondaries:
   # role: secondary
   # cluster: <cluster-name>
   

2. Check DNSZone reconciler logs:

   kubectl logs -n bindy-system deployment/bindy | grep "secondary"
   

3. Verify network connectivity:

   # Test AXFR from secondary to primary
   kubectl exec -n bindy-system deployment/secondary-dns -- \
     dig @primary-dns-service AXFR example.com
   

Recovery After Secondary Pod Restart:

When secondary pods are rescheduled and get new IPs: 1. Detection: Reconciler automatically detects IP change within 5-10 minutes (next reconciliation) 2. Update: Zones are deleted and recreated with new secondary IPs 3. Transfer: Zone transfers resume automatically with new IPs

Manual Trigger (if needed):

# Force reconciliation by updating zone annotation
kubectl annotate dnszone example-com -n bindy-system \
  reconcile.bindy.firestoned.io/trigger="$(date +%s)" --overwrite

Performance Issues

High Query Latency

Symptom: DNS queries taking too long

Diagnosis:

# Test query time
time dig @$SERVICE_IP example.com

# Check resource usage
kubectl top pods -n bindy-system -l instance=primary-dns

Solutions:

1. Increase resources:

   spec:
     resources:
       limits:
         cpu: "1000m"
         memory: "1Gi"
   

2. Add more replicas:

   spec:
     replicas: 3
   

3. Enable caching (if appropriate for your use case)

RBAC Issues

Forbidden Errors in Logs

Symptom: Operator logs show "Forbidden" errors

Diagnosis:

kubectl logs -n bindy-system deployment/bindy | grep Forbidden

# Check permissions
kubectl auth can-i create deployments \
  --as=system:serviceaccount:bindy-system:bindy \
  -n bindy-system

Solution:

# Reapply RBAC
kubectl apply -f deploy/rbac/

# Verify ClusterRoleBinding
kubectl get clusterrolebinding bindy-rolebinding -o yaml

# Restart operator
kubectl rollout restart deployment/bindy -n bindy-system

bindcar Sidecar Authentication (Mode B / TokenReview)

Sidecar crashloops: "refusing to start ... without real authentication"

Symptom: the operand pod's api container crashloops; its logs show: refusing to start: the API is bound to a non-loopback interface (0.0.0.0) without real authentication.

Cause: the bindcar image lacks the TokenReview auth path. bindy runs "Mode B" (no BIND_API_TOKEN), so the image must be built with --features k8s-token-review. Published images >= v0.7.2 include it; v0.7.0/v0.7.1 do not.

Solution: use ghcr.io/firestoned/bindcar:v0.7.2 or newer (the default). If you pin bindcarConfig.image, bump it to v0.7.2+.

Operator gets HTTP 401 from the bindcar API

Symptom: operator logs show persistent 401 Unauthorized from bindcar zone operations (a 401 is not retried), and zones never become Ready.

Diagnosis:

# Is the operand SA allowed to create tokenreviews?
kubectl auth can-i create tokenreviews.authentication.k8s.io \
  --as=system:serviceaccount:bindy-system:bind9

# Does the operator carry the bindcar-audience projected token?
kubectl get deploy/bindy -n bindy-system -o jsonpath='{.spec.template.spec.volumes[?(@.name=="bindcar-token")].projected.sources[0].serviceAccountToken.audience}'
# expect: bindcar

Common causes & fixes: - Missing tokenreview RBAC → kubectl apply -f deploy/operator/rbac/tokenreview-clusterrole.yaml -f deploy/operator/rbac/tokenreview-clusterrolebinding.yaml (add a subject per operand namespace). - Audience mismatch → the operator's projected token audience must equal the sidecar's BIND_TOKEN_AUDIENCES (bindcar). - Wrong allow-list → BIND_ALLOWED_SERVICE_ACCOUNTS must name the operator SA (system:serviceaccount:<ns>:bindy), not the operand bind9 SA.

See the bindcar 0.7.x migration guide and RBAC.

Next Steps

• Debugging Guide - Detailed debugging procedures
• FAQ - Frequently asked questions
• Logging - Log analysis