Skip to content

Metrics

Monitor performance and health metrics for Bindy DNS infrastructure.

Operator Metrics

Bindy exposes Prometheus-compatible metrics on port 8080 at /metrics. These metrics provide comprehensive observability into the operator's behavior and resource management.

Accessing Metrics

The metrics endpoint is exposed on all operator pods:

# Port forward to the operator
kubectl port-forward -n bindy-system deployment/bindy-operator 8080:8080

# View metrics
curl http://localhost:8080/metrics

Available Metrics

All metrics use the namespace prefix bindy_firestoned_io_.

Reconciliation Metrics

bindy_firestoned_io_reconciliations_total (Counter) Total number of reconciliation attempts by resource type and outcome.

Labels: - resource_type: Kind of resource (Bind9Cluster, Bind9Instance, DNSZone, ARecord, AAAARecord, TXTRecord, CNAMERecord, MXRecord, NSRecord, SRVRecord, CAARecord, PTRRecord) - status: Outcome (success, error)

# Reconciliation success rate
rate(bindy_firestoned_io_reconciliations_total{status="success"}[5m])

# Error rate by resource type
rate(bindy_firestoned_io_reconciliations_total{status="error"}[5m])

bindy_firestoned_io_reconciliation_duration_seconds (Histogram) Duration of reconciliation operations in seconds.

Labels: - resource_type: Kind of resource

Buckets: 0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0

# Average reconciliation duration
rate(bindy_firestoned_io_reconciliation_duration_seconds_sum[5m])
/ rate(bindy_firestoned_io_reconciliation_duration_seconds_count[5m])

# 95th percentile latency
histogram_quantile(0.95, bindy_firestoned_io_reconciliation_duration_seconds_bucket)

Resource Lifecycle Metrics

bindy_firestoned_io_resources_deleted_total (Counter) Total number of resources deleted (counted when the operator finishes cleaning up a DNSZone or DNS record that is being deleted).

Labels: - resource_type: Kind of resource

# Deletion rate by type
sum(rate(bindy_firestoned_io_resources_deleted_total[5m])) by (resource_type)

Secret Operation Metrics

bindy_firestoned_io_secret_operations_total (Counter) Every Secret operation bindy performs, one per audit event (Secret Audit Log, ADR-0024).

Labels: - operation: create, replace, patch, delete, read - purpose: rndc-key, rndc-key-mirror, rndc-key-source, bindcar-ca, scout-kubeconfig, serviceaccount-token - outcome: success, failure

# RNDC key loads or writes failing
sum by (operation) (rate(bindy_firestoned_io_secret_operations_total{purpose=~"rndc-key.*", outcome="failure"}[5m]))

Error Metrics

bindy_firestoned_io_errors_total (Counter) Total number of errors by resource type and category.

Labels: - resource_type: Kind of resource - error_type: Category (api_error, validation_error, network_error, timeout, reconcile_error)

# Error rate by type
rate(bindy_firestoned_io_errors_total[5m])

# Errors by resource type
sum(rate(bindy_firestoned_io_errors_total[5m])) by (resource_type)

Leader Election Metrics

bindy_firestoned_io_leader_elections_total (Counter) Total number of leader election events: acquired each time this replica takes the lease, lost when it loses it (the replica then drains its controllers and restarts as a follower). With leader election disabled no event is counted.

Labels: - status: Event type (acquired, lost)

bindy_firestoned_io_leader_status (Gauge) Current leader election status (1 = leader, 0 = follower). A replica reports 0 while it waits for the lease and 1 once it holds it. With leader election disabled the single replica reports 1.

Labels: - pod_name: The replica's identity (POD_NAME, else HOSTNAME)

# Current leader
bindy_firestoned_io_leader_status == 1

# Leader election rate
rate(bindy_firestoned_io_leader_elections_total[5m])

Kubernetes API Client Metrics

Recorded by the client middleware installed per ADR-0005; see the configuration guide for the QPS/burst settings they observe.

bindy_firestoned_io_kube_api_requests_total (Counter) Every Kubernetes API request issued by the operator's client.

Labels: - resource: Resource plural parsed from the request path (e.g., dnszones) - verb: Lowercase HTTP method (get, post, patch, ...) - status: success (2xx/3xx) or error. A request cut by the client-side request deadline (ADR-0014, BINDY_KUBE_REQUEST_TIMEOUT_SECS) counts as error when it timed out waiting for the response headers.

bindy_firestoned_io_kube_api_request_duration_seconds (Histogram) Request latency from dispatch to response headers. Time queued behind the client-side rate limiter is not included.

Labels: resource, verb. Buckets: 0.001, 0.01, 0.05, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0

bindy_firestoned_io_kube_api_rate_limit_hits_total (Counter) HTTP 429 responses from the API server. Under the client-side rate limiter this should stay at zero — a non-zero rate means the configured QPS/burst exceeds what the API server tolerates.

Labels: resource, verb.

bindy_firestoned_io_kube_api_retries_total (Counter) Retries of transient API failures (429/5xx/network) with exponential backoff.

Labels: - operation: Operation retried (e.g., get Bind9Cluster)

bindy_firestoned_io_kube_api_pagination_pages (Histogram) Pages fetched per paginated list operation (100 items per page).

Labels: resource. Buckets: 1, 2, 3, 5, 10, 25, 50, 100

# Operator API request rate by verb
sum by (verb) (rate(bindy_firestoned_io_kube_api_requests_total[5m]))

# Server-side throttling — should be zero; alert if sustained
rate(bindy_firestoned_io_kube_api_rate_limit_hits_total[5m]) > 0.1

# API retry rate — sustained retries indicate API server pressure
rate(bindy_firestoned_io_kube_api_retries_total[5m]) > 1.0

Suggested alert rules:

- alert: BindyHighAPIRateLimitHits
  expr: sum(rate(bindy_firestoned_io_kube_api_rate_limit_hits_total[5m])) > 0.1
  for: 5m
  annotations:
    summary: "Bindy hitting Kubernetes API server rate limits"
    description: "{{ $value }} HTTP 429 responses per second — lower BINDY_KUBE_QPS/BINDY_KUBE_BURST or raise API server priority"

- alert: BindyHighAPIRetryRate
  expr: sum(rate(bindy_firestoned_io_kube_api_retries_total[5m])) > 1.0
  for: 10m
  annotations:
    summary: "Bindy experiencing a high Kubernetes API retry rate"
    description: "{{ $value }} retries per second — check API server health and operator logs"

Prometheus Configuration

The operator deployment includes Prometheus scrape annotations:

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "8080"
  prometheus.io/path: "/metrics"

Prometheus will automatically discover and scrape these metrics if configured with Kubernetes service discovery.

Example Queries

# Reconciliation success rate (last 5 minutes)
sum(rate(bindy_firestoned_io_reconciliations_total{status="success"}[5m]))
/ sum(rate(bindy_firestoned_io_reconciliations_total[5m]))

# DNSZone reconciliation p95 latency
histogram_quantile(0.95,
  sum(rate(bindy_firestoned_io_reconciliation_duration_seconds_bucket{resource_type="DNSZone"}[5m])) by (le)
)

# Error rate by resource type (last hour)
topk(10,
  sum(rate(bindy_firestoned_io_errors_total[1h])) by (resource_type)
)

Grafana Dashboard

A ready-to-import dashboard for the operator ships in the repository at docs/grafana-dashboard.json.

Import it from Grafana → Dashboards → New → Import, upload the JSON, and select your Prometheus datasource for the DS_PROMETHEUS variable.

It includes panels for:

  1. Reconciliation Rate - Total reconciliations/sec by resource type
  2. Reconciliation Latency - P50, P95, P99 latencies
  3. Error Rate - Errors/sec by resource type and error category
  4. Leader Status - Current leader pod and election events
  5. Resource Deletions - Deletion rate by resource type

A companion dashboard for the bindcar sidecar ships at docs/grafana-dashboard-bindcar.json.

Sidecar (bindcar) Metrics

The bindcar sidecar that runs alongside named exposes its own Prometheus metrics on /metrics (default port 8080, unauthenticated), using the bindcar_ prefix. Scrape it with the same prometheus.io/scrape annotations as the operator.

Metric Type Labels Description
bindcar_http_requests_total Counter method, path, status HTTP requests processed by the REST API
bindcar_http_request_duration_seconds Histogram method, path HTTP request latency
bindcar_zone_operations_total Counter operation, result Zone and record operations (record ops use the record_* operation prefix)
bindcar_rndc_commands_total Counter command, result rndc commands executed (nsupdate uses the nsupdate_* command prefix)
bindcar_rndc_command_duration_seconds Histogram command rndc/nsupdate command latency
bindcar_zones_managed Gauge — Zones currently managed by this instance
bindcar_rate_limit_requests_total Counter result (allowed, rejected) Rate-limit decisions
bindcar_app_info Counter version Build/version info 
# Request rate by HTTP status
sum(rate(bindcar_http_requests_total[5m])) by (status)

# API p95 latency
histogram_quantile(0.95, sum(rate(bindcar_http_request_duration_seconds_bucket[5m])) by (le))

# Failed rndc commands
sum(rate(bindcar_rndc_commands_total{result="error"}[5m])) by (command)

# Rate-limit rejections
sum(rate(bindcar_rate_limit_requests_total{result="rejected"}[5m]))

Import the sidecar dashboard from docs/grafana-dashboard-bindcar.json.

Resource Metrics

Pod Metrics

View CPU and memory usage:

# All DNS pods
kubectl top pods -n bindy-system

# Specific instance
kubectl top pods -n bindy-system -l instance=primary-dns

# Sort by CPU
kubectl top pods -n bindy-system --sort-by=cpu

# Sort by memory
kubectl top pods -n bindy-system --sort-by=memory

Node Metrics

# Node resource usage
kubectl top nodes

# Detailed node info
kubectl describe node <node-name>

DNS Query Metrics

Using BIND9 Statistics

Enable BIND9 statistics channel (future enhancement):

spec:
  config:
    statisticsChannels:
      - address: "127.0.0.1"
        port: 8053

Query Counters

Monitor query rate and types: - Total queries received - Queries by record type (A, AAAA, MX, etc.) - Successful vs failed queries - NXDOMAIN responses

Performance Metrics

Query Latency

Measure DNS query response time:

# Test query latency
time dig @<dns-server-ip> example.com

# Multiple queries for average
for i in {1..10}; do time dig @<dns-server-ip> example.com +short; done

Zone Transfer Metrics

Monitor zone transfer performance: - Transfer duration - Transfer size - Transfer failures - Lag between primary and secondary

Kubernetes Metrics

Resource Utilization

# View resource requests vs limits
kubectl describe pod -n bindy-system <pod-name> | grep -A5 "Limits:\|Requests:"

Pod Health

# Pod status and restarts
kubectl get pods -n bindy-system -o wide

# Events
kubectl get events -n bindy-system --sort-by='.lastTimestamp'

Prometheus Integration

BIND9 Exporter

Deploy bind_exporter as sidecar (future enhancement):

containers:
- name: bind-exporter
  image: prometheuscommunity/bind-exporter:latest
  args:
    - "--bind.stats-url=http://localhost:8053"
  ports:
    - name: metrics
      containerPort: 9119

Service Monitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: bindy-metrics
spec:
  selector:
    matchLabels:
      app: bind9
  endpoints:
  - port: metrics
    interval: 30s

Key Metrics to Monitor

  1. Query Rate - Queries per second
  2. Query Latency - Response time
  3. Error Rate - Failed queries percentage
  4. Cache Hit Ratio - Cache effectiveness
  5. Zone Transfer Status - Success/failure of transfers
  6. Resource Usage - CPU and memory utilization
  7. Pod Health - Running vs desired replicas

Grafana Dashboards

Create dashboards for:

DNS Overview

  • Total query rate
  • Average latency
  • Error rate
  • Top queried domains

Instance Health

  • Pod status
  • CPU/memory usage
  • Restart count
  • Network I/O

Zone Management

  • Zones count
  • Records per zone
  • Zone transfer status
  • Serial numbers

Alerting Thresholds

Recommended alert thresholds:

Metric Warning Critical
CPU Usage > 70% > 90%
Memory Usage > 70% > 90%
Query Latency > 100ms > 500ms
Error Rate > 1% > 5%
Pod Restarts > 3/hour > 10/hour

Best Practices

  1. Baseline metrics - Establish normal operating ranges
  2. Set appropriate alerts - Avoid alert fatigue
  3. Monitor trends - Look for gradual degradation
  4. Capacity planning - Use metrics to plan scaling
  5. Regular review - Review dashboards weekly