Osmosis Monitoring Setup

Chain ID:

osmo-test-5

Current Node Version:

v29.0.0-rc1

Monitoring Stack

This guide helps you set up comprehensive monitoring for your Osmosis validator node using Prometheus and Grafana.

Prerequisites

• Docker and Docker Compose installed
• Osmosis node running with metrics enabled
• Ports 9090 (Prometheus) and 3000 (Grafana) available

Step 1: Enable Metrics in Osmosis Node

Edit your config.toml file:

config.toml

# Enable metrics
prometheus = true
prometheus_listen_addr = "26661"

Restart your osmosisd service:

sudo systemctl restart osmosisd

Step 2: Create Monitoring Directory

mkdir -p $HOME/osmosis-monitoring
cd $HOME/osmosis-monitoring

Step 3: Docker Compose Configuration

Create docker-compose.yml:

docker-compose.yml

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: osmosis-prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alert_rules.yml:/etc/prometheus/alert_rules.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-lifecycle'
      - '--web.enable-admin-api'
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: osmosis-grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin123
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
      - ./grafana/datasources:/etc/grafana/provisioning/datasources
    restart: unless-stopped

  node-exporter:
    image: prom/node-exporter:latest
    container_name: osmosis-node-exporter
    ports:
      - "9100:9100"
    command:
      - '--path.rootfs=/host'
    volumes:
      - '/:/host:ro,rslave'
    restart: unless-stopped

volumes:
  prometheus_data:
  grafana_data:

Step 4: Prometheus Configuration

Create prometheus.yml:

prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alert_rules.yml"

scrape_configs:
  - job_name: 'osmosis-validator'
    static_configs:
      - targets: ['host.docker.internal:26661']
    scrape_interval: 10s
    metrics_path: /metrics

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
    scrape_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: []

Step 5: Alert Rules

Create alert_rules.yml:

alert_rules.yml

groups:
  - name: osmosis-validator-alerts
    rules:
      - alert: ValidatorDown
        expr: up{job="osmosis-validator"} == 0
        for: 1m
        labels:
          severity: critical
          chain: Osmosis
        annotations:
          summary: "Osmosis validator is down"
          description: "Osmosis validator has been down for more than 1 minute"

      - alert: NodeNotSyncing
        expr: tendermint_consensus_height - tendermint_consensus_latest_block_height > 10
        for: 5m
        labels:
          severity: warning
          chain: Osmosis
        annotations:
          summary: "Osmosis node is not syncing"
          description: "Osmosis node is behind by {{ $value }} blocks"

      - alert: MissedBlocks
        expr: tendermint_consensus_validator_missed_blocks > 5
        for: 2m
        labels:
          severity: warning
          chain: Osmosis
        annotations:
          summary: "Osmosis validator missing blocks"
          description: "Osmosis validator has missed {{ $value }} blocks"

      - alert: HighMemoryUsage
        expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes > 0.9
        for: 5m
        labels:
          severity: warning
          chain: Osmosis
        annotations:
          summary: "High memory usage on Osmosis validator"
          description: "Memory usage is above 90%"

      - alert: DiskSpaceLow
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.1
        for: 1m
        labels:
          severity: critical
          chain: Osmosis
        annotations:
          summary: "Low disk space on Osmosis validator"
          description: "Disk space is below 10%"

Step 6: Grafana Datasource Configuration

Create grafana/datasources/prometheus.yml:

mkdir -p grafana/datasources grafana/dashboards

grafana/datasources/prometheus.yml

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true

Step 7: Launch Monitoring Stack

# Start the monitoring stack
docker-compose up -d

# Check status
docker-compose ps

# View logs
docker-compose logs -f

Step 8: Access Dashboards

• Grafana Dashboard: http://localhost:3000

  • Username: admin
  • Password: admin123

• Prometheus: http://localhost:9090

Pre-built Dashboard

Import our Osmosis validator dashboard from NODERS:

🔗 Dashboard JSON: https://snapshots.noders.services/monitoring/cosmos-dashboard.json

How to Import:

1. Open Grafana at http://localhost:3000
2. Click "+" → "Import"
3. Paste the URL: https://snapshots.noders.services/monitoring/cosmos-dashboard.json
4. Click "Load" and configure data sources
5. Select your Prometheus datasource and save

Dashboard Features:

• Validator status and uptime
• Block height and sync status
• Missed blocks tracking
• System resources (CPU, Memory, Disk)
• Network metrics and TCP connections
• Peer connections and network health
• Hardware monitoring with node-exporter
• Custom alerts and thresholds

Useful Queries

Validator Status:

up{job="osmosis-validator"}

Block Height:

tendermint_consensus_height

Missed Blocks:

tendermint_consensus_validator_missed_blocks

Memory Usage:

(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100

Troubleshooting

Metrics not showing up:

# Check if metrics endpoint is accessible
curl http://localhost:26661/metrics

# Check Prometheus targets
curl http://localhost:9090/api/v1/targets

# Check node-exporter metrics
curl http://localhost:9100/metrics

Grafana connection issues:

# Check Grafana logs
docker-compose logs grafana

# Restart Grafana
docker-compose restart grafana

Advanced Configuration

Add more exporters:

• Node Exporter: System metrics
• Process Exporter: osmosisd process metrics
• Blackbox Exporter: Endpoint monitoring

Custom Alerts: Edit alert_rules.yml to add chain-specific alert rules based on your validator requirements.

Pro Tip

Set up alerting notifications (Slack, Discord, Email) by configuring Alertmanager for real-time validator monitoring.

Cleanup

To stop and remove the monitoring stack:

docker-compose down -v