Skip to main content
Airchains Monitoring Setup

Airchains Monitoring Setup

Chain ID:

varanasi-1

Current Node Version:

v0.3.2

Monitoring Stack​

This guide helps you set up comprehensive monitoring for your Airchains validator node using Prometheus and Grafana.

Prerequisites​

  • Docker and Docker Compose installed
  • Airchains node running with metrics enabled
  • Ports 9090 (Prometheus) and 3000 (Grafana) available

Step 1: Enable Metrics in Airchains Node​

Edit your config.toml file:

config.toml
# Enable metrics
prometheus = true
prometheus_listen_addr = "26661"

Restart your junctiond service:

sudo systemctl restart junctiond

Step 2: Create Monitoring Directory​

mkdir -p $HOME/airchains-monitoring
cd $HOME/airchains-monitoring

Step 3: Docker Compose Configuration​

Create docker-compose.yml:

docker-compose.yml
version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: airchains-prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alert_rules.yml:/etc/prometheus/alert_rules.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-lifecycle'
      - '--web.enable-admin-api'
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: airchains-grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin123
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
      - ./grafana/datasources:/etc/grafana/provisioning/datasources
    restart: unless-stopped

  node-exporter:
    image: prom/node-exporter:latest
    container_name: airchains-node-exporter
    ports:
      - "9100:9100"
    command:
      - '--path.rootfs=/host'
    volumes:
      - '/:/host:ro,rslave'
    restart: unless-stopped

volumes:
  prometheus_data:
  grafana_data:

Step 4: Prometheus Configuration​

Create prometheus.yml:

prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alert_rules.yml"

scrape_configs:
  - job_name: 'airchains-validator'
    static_configs:
      - targets: ['host.docker.internal:26661']
    scrape_interval: 10s
    metrics_path: /metrics

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
    scrape_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: []

Step 5: Alert Rules​

Create alert_rules.yml:

alert_rules.yml
groups:
  - name: airchains-validator-alerts
    rules:
      - alert: ValidatorDown
        expr: up{job="airchains-validator"} == 0
        for: 1m
        labels:
          severity: critical
          chain: Airchains
        annotations:
          summary: "Airchains validator is down"
          description: "Airchains validator has been down for more than 1 minute"

      - alert: NodeNotSyncing
        expr: tendermint_consensus_height - tendermint_consensus_latest_block_height > 10
        for: 5m
        labels:
          severity: warning
          chain: Airchains
        annotations:
          summary: "Airchains node is not syncing"
          description: "Airchains node is behind by {{ $value }} blocks"

      - alert: MissedBlocks
        expr: tendermint_consensus_validator_missed_blocks > 5
        for: 2m
        labels:
          severity: warning
          chain: Airchains
        annotations:
          summary: "Airchains validator missing blocks"
          description: "Airchains validator has missed {{ $value }} blocks"

      - alert: HighMemoryUsage
        expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes > 0.9
        for: 5m
        labels:
          severity: warning
          chain: Airchains
        annotations:
          summary: "High memory usage on Airchains validator"
          description: "Memory usage is above 90%"

      - alert: DiskSpaceLow
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.1
        for: 1m
        labels:
          severity: critical
          chain: Airchains
        annotations:
          summary: "Low disk space on Airchains validator"
          description: "Disk space is below 10%"

Step 6: Grafana Datasource Configuration​

Create grafana/datasources/prometheus.yml:

mkdir -p grafana/datasources grafana/dashboards
grafana/datasources/prometheus.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true

Step 7: Launch Monitoring Stack​

# Start the monitoring stack
docker-compose up -d

# Check status
docker-compose ps

# View logs
docker-compose logs -f

Step 8: Access Dashboards​

Pre-built Dashboard​

Import our Airchains validator dashboard from NODERS:

🔗 Dashboard JSON: https://snapshots.noders.services/monitoring/cosmos-dashboard.json

How to Import:

  1. Open Grafana at http://localhost:3000
  2. Click "+" → "Import"
  3. Paste the URL: https://snapshots.noders.services/monitoring/cosmos-dashboard.json
  4. Click "Load" and configure data sources
  5. Select your Prometheus datasource and save

Dashboard Features:

  • Validator status and uptime
  • Block height and sync status
  • Missed blocks tracking
  • System resources (CPU, Memory, Disk)
  • Network metrics and TCP connections
  • Peer connections and network health
  • Hardware monitoring with node-exporter
  • Custom alerts and thresholds

Useful Queries​

Validator Status:

up{job="airchains-validator"}

Block Height:

tendermint_consensus_height

Missed Blocks:

tendermint_consensus_validator_missed_blocks

Memory Usage:

(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100

Troubleshooting​

Metrics not showing up:

# Check if metrics endpoint is accessible
curl http://localhost:26661/metrics

# Check Prometheus targets
curl http://localhost:9090/api/v1/targets

# Check node-exporter metrics
curl http://localhost:9100/metrics

Grafana connection issues:

# Check Grafana logs
docker-compose logs grafana

# Restart Grafana
docker-compose restart grafana

Advanced Configuration​

Add more exporters:

  • Node Exporter: System metrics
  • Process Exporter: junctiond process metrics
  • Blackbox Exporter: Endpoint monitoring

Custom Alerts: Edit alert_rules.yml to add chain-specific alert rules based on your validator requirements.

Pro Tip

Set up alerting notifications (Slack, Discord, Email) by configuring Alertmanager for real-time validator monitoring.

Cleanup​

To stop and remove the monitoring stack:

docker-compose down -v