Skip to content

High Availability

Design and implement highly available DNS infrastructure with Bindy.

Overview

High availability (HA) DNS ensures continuous DNS service even during: - Pod failures - Node failures - Availability zone outages - Regional outages - Planned maintenance

HA Architecture Components

1. Multiple Replicas

Run multiple replicas of each Bind9Instance:

apiVersion: bindy.firestoned.io/v1beta1
kind: Bind9Instance
metadata:
  name: primary-dns
spec:
  replicas: 3  # Multiple replicas for pod-level HA

Benefits: - Survives pod crashes - Load distribution - Zero-downtime updates

2. Multiple Instances

Deploy separate primary and secondary instances:

# Primary instance
apiVersion: bindy.firestoned.io/v1beta1
kind: Bind9Instance
metadata:
  name: primary-dns
  labels:
    dns-role: primary
spec:
  replicas: 2
---
# Secondary instance  
apiVersion: bindy.firestoned.io/v1beta1
kind: Bind9Instance
metadata:
  name: secondary-dns
  labels:
    dns-role: secondary
spec:
  replicas: 2

Benefits: - Role separation - Independent scaling - Failover capability

3. Geographic Distribution

Deploy instances across multiple regions:

# US East primary
apiVersion: bindy.firestoned.io/v1beta1
kind: Bind9Instance
metadata:
  name: primary-us-east
  labels:
    dns-role: primary
    region: us-east-1
spec:
  replicas: 2
---
# US West secondary
apiVersion: bindy.firestoned.io/v1beta1
kind: Bind9Instance
metadata:
  name: secondary-us-west
  labels:
    dns-role: secondary
    region: us-west-2
spec:
  replicas: 2
---
# EU secondary
apiVersion: bindy.firestoned.io/v1beta1
kind: Bind9Instance
metadata:
  name: secondary-eu-west
  labels:
    dns-role: secondary
    region: eu-west-1
spec:
  replicas: 2

Benefits: - Regional failure tolerance - Lower latency for global users - Regulatory compliance (data locality)

HA Patterns

Pattern 1: Active-Passive

One active primary, multiple passive secondaries:

graph LR primary["Primary<br/>(Active)<br/>us-east-1"] sec1["Secondary<br/>(Passive)<br/>us-west-2"] sec2["Secondary<br/>(Passive)<br/>eu-west-1"] clients["Clients query any"] primary -->|AXFR| sec1 sec1 -->|AXFR| sec2 primary --> clients sec1 --> clients sec2 --> clients style primary fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px style sec1 fill:#e1f5ff,stroke:#01579b,stroke-width:2px style sec2 fill:#e1f5ff,stroke:#01579b,stroke-width:2px style clients fill:#fff9c4,stroke:#f57f17,stroke-width:2px
  • Updates go to primary only
  • Secondaries receive via zone transfer
  • Clients query any available instance

Pattern 2: Multi-Primary

Multiple primaries in different regions:

graph LR primary1["Primary<br/>(zone-a)<br/>us-east-1"] primary2["Primary<br/>(zone-b)<br/>eu-west-1"] primary1 <-->|Sync| primary2 style primary1 fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px style primary2 fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px
  • Different zones on different primaries
  • Geographic distribution of updates
  • Careful coordination required

Pattern 3: Anycast

Same IP announced from multiple locations:

graph TB client["Client Query (192.0.2.53)"] dns_us["DNS<br/>US"] dns_eu["DNS<br/>EU"] dns_apac["DNS<br/>APAC"] client --> dns_us client --> dns_eu client --> dns_apac style client fill:#fff9c4,stroke:#f57f17,stroke-width:2px style dns_us fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px style dns_eu fill:#e1f5ff,stroke:#01579b,stroke-width:2px style dns_apac fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
  • Requires BGP routing
  • Lowest latency routing
  • Automatic failover

Pod-Level HA

Anti-Affinity

Spread pods across nodes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: primary-dns
spec:
  replicas: 3
  template:
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: instance
                  operator: In
                  values:
                  - primary-dns
              topologyKey: kubernetes.io/hostname

Topology Spread

Distribute across availability zones:

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        instance: primary-dns

Service-Level HA

Liveness and Readiness Probes

Ensure only healthy pods serve traffic:

livenessProbe:
  exec:
    command: ["dig", "@localhost", "version.bind", "txt", "chaos"]
  initialDelaySeconds: 30
  periodSeconds: 10

readinessProbe:
  exec:
    command: ["dig", "@localhost", "version.bind", "txt", "chaos"]
  initialDelaySeconds: 5
  periodSeconds: 5

Rollouts: the Zones-Loaded Readiness Gate

A probe tells Kubernetes that named and bindcar are up, not that the pod holds any zone. BIND9 pods keep zone data on emptyDir volumes, so a replacement pod starts empty. bindy therefore adds a readiness gate to every BIND9 pod, readinessGates: [{conditionType: bindy.firestoned.io/zones-loaded}] (ADR-0017):

  • The operator loads every live zone of the pod's instance onto the new pod (and, on a primary, replays every record), then sets the condition True. Only then does the pod become Ready and join its Service.
  • A one-replica BIND9 Deployment (each cluster-managed instance) rolls with the default RollingUpdate, which rounds to maxSurge 1 and maxUnavailable 0, so the old pod keeps serving until the new one is Ready: a rollout, including one that rolls every primary at once after an operator upgrade, never leaves a nameserver answering REFUSED.
  • kubectl rollout status completes when the new pod's zones are loaded.
  • If the operator is down, new pods wait and the old ones keep serving.
  • A zone that cannot be loaded blocks a new pod only while another pod of the instance still serves it; see Troubleshooting.
  • The old pod's gate closes at the start of its termination. When a BIND9 pod is deleted, the operator sets its gate back to False (PodTerminating) at once. The pod's Ready (and its endpoint's serving) follow only when the kubelet next syncs the pod's status, measured at about 18 s on v0.8.0-rc.6, so the flip does not by itself move traffic. What keeps the handover short is that the new pod is Ready before the old one is deleted and that instances roll one at a time (below): on rc.6 a staggered rollout of three instances lost 2 of 136 probe queries, against 9 to 12 s of outage on both LoadBalancer IPs when rc.5 rolled them all at once.

Staggered Rollouts Across Instances

A change of a cluster's shared configuration, an image change at the cluster level, or a bindy upgrade that changes the rendered configuration or the pod template would roll every instance at the same moment. bindy rolls them one at a time instead (ADR-0018): an instance applies a pod-template change only while no instance that serves a zone in common with it, or belongs to the same Bind9Cluster / ClusterBind9Provider, is mid-rollout. The others wait with Rollout=False, reason: RolloutQueued and keep serving. Replica changes and new instances are never delayed, and a rollout stuck past its Deployment's progressDeadlineSeconds (600 s by default) stops blocking the others.

LoadBalancer Services: externalTrafficPolicy and MetalLB Layer 2

How fast traffic leaves a terminating pod depends on the Service in front of it (set per role in the Bind9Cluster spec.primary.service / spec.secondary.service).

externalTrafficPolicy: Cluster (default) externalTrafficPolicy: Local
Who receives traffic Any node; kube-proxy forwards to any Ready pod Only a node with a Ready local pod; kube-proxy sends to that node's pods only
Client source IP Rewritten (SNAT) Preserved
Handover when a pod terminates No gap: once the old pod is not Ready, every node forwards to the new one MetalLB moves the announcement to the new pod's node; until clients take the gratuitous ARP ("a few seconds", longer for clients with buggy ARP handling), packets to the old node are dropped

With Local, a handover that takes the last Ready pod off the announcing node therefore still has a short gap that is MetalLB's own reaction time. Staggered rollouts keep the zone's other nameservers answering during it. To shrink or remove the gap:

  • use externalTrafficPolicy: Cluster where the client source IP is not needed (for example when no ACL or rate limit keys on it): this removes the gap;
  • or run more than one replica per instance, spread over nodes: only the replacement of the pod on the announcing node moves the announcement, and the instance keeps answering through its other pods inside the cluster. This makes the gap rarer, it does not remove it.

Pod Disruption Budgets

Limit concurrent disruptions:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: primary-dns-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      instance: primary-dns

Monitoring HA

Check Instance Distribution

# View instances across regions
kubectl get bind9instances -A -L region

# View pod distribution
kubectl get pods -n bindy-system -o wide

# Check zone spread
kubectl get pods -n bindy-system \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,ZONE:.spec.nodeSelector

Test Failover

# Simulate pod failure
kubectl delete pod -n bindy-system <pod-name>

# Verify automatic recovery
kubectl get pods -n bindy-system -w

# Test DNS during failover
while true; do dig @$SERVICE_IP example.com +short; sleep 1; done

Disaster Recovery

Backup Strategy

# Regular backups of all CRDs
kubectl get bind9instances,dnszones,arecords,aaaarecords,cnamerecords,mxrecords,txtrecords,nsrecords,srvrecords,caarecords,ptrrecords \
  -A -o yaml > backup-$(date +%Y%m%d).yaml

Recovery Procedures

  1. Single Pod Failure - Kubernetes automatically recreates
  2. Instance Failure - Clients fail over to other instances
  3. Regional Failure - Zone data available from other regions
  4. Complete Loss - Restore from backup
# Restore from backup
kubectl apply -f backup-20241126.yaml

Operator High Availability

The Bindy operator itself can run in high availability mode with automatic leader election. This ensures continuous DNS management even if operator pods fail.

Leader Election

Multiple operator instances use Kubernetes Lease objects for distributed leader election:

graph TB op1["Operator<br/>Instance 1<br/>(Leader)"] op2["Operator<br/>Instance 2<br/>(Standby)"] op3["Operator<br/>Instance 3<br/>(Standby)"] lease["Kubernetes API<br/>Lease Object"] op1 --> lease op2 --> lease op3 --> lease style op1 fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px style op2 fill:#e1f5ff,stroke:#01579b,stroke-width:2px style op3 fill:#e1f5ff,stroke:#01579b,stroke-width:2px style lease fill:#fff9c4,stroke:#f57f17,stroke-width:2px

How it works: 1. All operator instances attempt to acquire the lease 2. One instance becomes the leader and starts reconciling resources 3. Standby instances wait and monitor the lease 4. If the leader fails, a standby automatically takes over (~15 seconds)

HA Operator Deployment

Deploy multiple operator replicas with leader election enabled:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bindy
  namespace: bindy-system
spec:
  replicas: 3  # Run 3 instances for HA
  selector:
    matchLabels:
      app: bindy
  template:
    metadata:
      labels:
        app: bindy
    spec:
      serviceAccountName: bindy
      containers:
      - name: operator
        image: ghcr.io/firestoned/bindy:latest
        env:
        # Leader election configuration
        - name: ENABLE_LEADER_ELECTION
          value: "true"
        - name: LEASE_NAME
          value: "bindy-leader"
        - name: LEASE_NAMESPACE
          value: "bindy-system"
        - name: LEASE_DURATION_SECONDS
          value: "15"
        - name: LEASE_RENEW_DEADLINE_SECONDS
          value: "10"
        - name: LEASE_RETRY_PERIOD_SECONDS
          value: "2"
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name

Configuration Options

Environment variables for leader election:

Variable Default Description
ENABLE_LEADER_ELECTION true Enable/disable leader election
LEASE_NAME bindy-leader Name of the Lease resource
LEASE_NAMESPACE bindy-system Namespace for the Lease
LEASE_DURATION_SECONDS 15 How long leader holds lease
LEASE_RENEW_DEADLINE_SECONDS 10 Leader must renew before this
LEASE_RETRY_PERIOD_SECONDS 2 How often to attempt lease acquisition
POD_NAME $HOSTNAME Unique identity for this operator instance

Monitoring Leader Election

Check which operator instance is the current leader:

# View the lease object
kubectl get lease -n bindy-system bindy-leader -o yaml

# Output shows current leader
spec:
  holderIdentity: bindy-7d8f9c5b4d-x7k2m  # Current leader pod
  leaseDurationSeconds: 15
  renewTime: "2025-11-30T12:34:56Z"

Monitor operator logs to see leadership changes:

# Watch operator logs
kubectl logs -n bindy-system deployment/bindy -f

# Look for leadership events
INFO Attempting to acquire lease bindy-leader
INFO Lease acquired, this instance is now the leader
INFO Starting all operators
WARN Leadership lost! Stopping all operators...
INFO Lease acquired, this instance is now the leader

Failover Testing

Test automatic failover:

# Find current leader
LEADER=$(kubectl get lease -n bindy-system bindy-leader -o jsonpath='{.spec.holderIdentity}')
echo "Current leader: $LEADER"

# Delete the leader pod
kubectl delete pod -n bindy-system $LEADER

# Watch for new leader election (typically ~15 seconds)
kubectl get lease -n bindy-system bindy-leader -w

# Verify DNS operations continue uninterrupted
kubectl get bind9instances -A

RBAC Requirements

Leader election requires additional permissions in the operator's Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: bindy
  namespace: bindy-system
rules:
# Leases for leader election
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get", "create", "update", "patch"]

Troubleshooting

Operator not reconciling resources:

# Check which instance is leader
kubectl get lease -n bindy-system bindy-leader -o jsonpath='{.spec.holderIdentity}'

# Verify that pod exists and is running
kubectl get pods -n bindy-system

# Check operator logs
kubectl logs -n bindy-system deployment/bindy -f

Multiple operators reconciling (split brain):

This should never happen with proper leader election. If you suspect it:

# Check lease configuration
kubectl get lease -n bindy-system bindy-leader -o yaml

# Verify all operators use the same LEASE_NAME
kubectl get deployment -n bindy-system bindy -o yaml | grep LEASE_NAME

# Force lease release (recreate it)
kubectl delete lease -n bindy-system bindy-leader

Leader election disabled but multiple replicas running:

This will cause conflicts. Either: 1. Enable leader election: Set ENABLE_LEADER_ELECTION=true 2. Or run single replica: kubectl scale deployment bindy --replicas=1

Performance Impact

Leader election adds minimal overhead: - Failover time: ~15 seconds (configurable via LEASE_DURATION_SECONDS) - Network traffic: 1 lease renewal every 2 seconds from leader only - CPU/Memory: Negligible (<1% increase)

Best Practices

  1. Run 3+ Operator Replicas - For operator HA with leader election
  2. Run 3+ DNS Instance Replicas - Odd numbers for quorum
  3. Multi-AZ Deployment - Spread across availability zones
  4. Geographic Redundancy - At least 2 regions for critical zones
  5. Monitor Continuously - Alert on degraded HA
  6. Test Failover - Regular disaster recovery drills (both operator and DNS instances)
  7. Automate Recovery - Use Kubernetes self-healing
  8. Document Procedures - Runbooks for incidents
  9. Enable Leader Election - Always run operator with ENABLE_LEADER_ELECTION=true in production
  10. Monitor Lease Health - Alert if lease ownership changes frequently (indicates instability)

Next Steps