Monitoring Setup

This guide covers setting up Prometheus and Grafana for bridge monitoring.

Overview

flowchart LR
    subgraph Relayers
        R1[Relayer 1 :9080]
        R2[Relayer 2 :9080]
        R3[Relayer 3 :9080]
    end

    subgraph Monitoring
        Prometheus[Prometheus :9090]
        Grafana[Grafana :3000]
        Alertmanager[Alertmanager :9093]
    end

    R1 --> Prometheus
    R2 --> Prometheus
    R3 --> Prometheus
    Prometheus --> Grafana
    Prometheus --> Alertmanager

Metrics Exposed

Each relayer exposes Prometheus metrics on port 9080.

Bridge Metrics

Relayer counters are per process: they reset on restart and every relayer counts the same transfer. Use them for rates and health, not for totals.

Bridge totals

Authoritative transfer counts and volumes come from the API indexer, which stores each canonical transfer once. Read them from GET /api/v1/stats/token-flow (exact amounts as decimal strings, with a per-deployment coverage flag that marks partial history) or from the API metrics listener as bridge_api_token_flow_*. Current escrow balances and wrapped supply are separate contract reads (/api/v1/tokens?includeLiquidity=true).

Independent Verification Metrics

Announcement discovery diagnostics

Connected bootstrap peers do not prove announcement discovery is converging. If Deferred trust for announced endpoint pending additional confirmations persists, compare worker, consumerKey, announcementKey, confirmationSource, announcementTimestamp, confirmations and required. Debug-level Recorded announced endpoint confirmation also records the transition to two. The keys are hashes, not endpoint URLs; distinct receiving ports have distinct consumer keys. These logs describe accepted observations, not every received packet.

Two distinct authorized sources are required per worker. Evidence expires after ten minutes without an accepted recording; accepted repeat-source observations renew that idle timer but never increase the count. Do not lower the threshold or assume waiting ten minutes establishes trust. Before the per-consumer correction, a fresh announcement arriving through a different source first could converge by chance, while repeated direct-first rounds stayed deferred.

Discovery bounds each worker to 100 concurrent announcement handlers and 500 endpoint confirmation records (at most two sources each). At capacity, new work/evidence is not admitted. The shared gossip cache retains up to 200 keys for six minutes and suppresses new gossip when full rather than evicting live keys. Each receiving server retains up to 10,000 announcement replay records and rejects excess entries until expiry. These overload limits do not lower trust requirements; fresh later announcements can retry as capacity becomes available.

Casper-to-EVM gateway delivery recovery

ERC-7786 gateway deliveries retain their attempt budget and retry deadline across restarts. Before broadcast, the relayer durably records the signed transaction and reserves one of the ten attempts. A failed state write prevents broadcast. Source, header, simulation and transaction-preparation reads that fail before broadcast do not consume attempts.

If acceptance or execution is unknown, the same transaction remains pending and counted. The signed transaction and its nonce are saved with the message. Nonces are allocated under a short per-chain lock, so concurrent deliveries never share one, and they broadcast in parallel. A send the process never completed, or the node rejected, is re-sent unchanged; one unmined for three minutes is replaced at the same nonce with a higher fee, and only one of its hashes can ever run. When the account's mined nonce has passed it, none of its hashes has a receipt, and the gateway shows the message undelivered, on two passes 30 seconds apart, the record is cleared and counts as a failed attempt; the message is signed afresh, or parks if that was the tenth. Attempts are never given back. A gap at the account's next nonce that blocks later deliveries is filled with a zero-value self-transfer. A missing receipt, failed read or unreadable gateway is never proof of loss. A delivery stuck behind a transaction that keeps failing to mine delays the deliveries queued after it on that chain; investigate it with the recorded hashes. Do not delete state or repeatedly restart to force another send. A confirmed failure counts once; delivery proof settles the message. Exhausted or permanently refused gateway messages remain durable and are not revived by restart.

EVM-to-Casper gateway delivery recovery

ERC-7786 gateway deliveries retain their attempt budget and retry deadline across restarts. Before broadcast, the relayer durably records the signed transaction and reserves one of the ten attempts. A failed state write prevents broadcast. Source, header, simulation and transaction-preparation reads that fail before broadcast do not consume attempts.

If acceptance or execution is unknown, the same transaction remains pending and counted. The signed transaction is saved with the message, so a send the process never completed, or the node rejected, is re-sent unchanged (same hash: it runs at most once). When the node explicitly reports the transaction unknown more than five minutes past its 30-minute validity window and the gateway shows the message undelivered, the record is cleared and counts as a failed attempt; the message is signed afresh, or parks if that was the tenth. Attempts are never given back. A timeout, failed read or unreadable gateway never clears the record; the relayer keeps waiting. Do not delete state or repeatedly restart to force another send. A confirmed failure counts once; delivery proof settles the message. Exhausted or permanently refused gateway messages remain durable and are not revived by restart.

P2P Metrics

Database Metrics

Prometheus Setup

prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
 - static_configs:
 - targets:
 - alertmanager:9093

rule_files:
 - "/etc/prometheus/alert_rules.yml"

scrape_configs:
 - job_name: 'bridge-relayers'
    static_configs:
 - targets:
 - 'relayer-1:9080'
 - 'relayer-2:9080'
 - 'relayer-3:9080'
 - 'relayer-4:9080'
 - 'relayer-5:9080'
    relabel_configs:
 - source_labels: [__address__]
        regex: 'relayer-(\d+):.*'
        target_label: relayer_id
        replacement: '$1'

 - job_name: 'node-exporter'
    static_configs:
 - targets:
 - 'relayer-1:9100'
 - 'relayer-2:9100'
 - 'relayer-3:9100'
 - 'relayer-4:9100'
 - 'relayer-5:9100'

Start Prometheus

# Docker
docker run -d \
 --name prometheus \
 -p 9090:9090 \
 -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

# Or with docker-compose
docker-compose up -d prometheus

Alert Rules

alert_rules.yml

groups:
 - name: bridge_alerts
    interval: 30s
    rules:
      # Relayer down
 - alert: RelayerDown
        expr: up{job="bridge-relayers"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Relayer {{ $labels.instance }} is down"
          description: "Relayer has been unreachable for 2 minutes"

      # High pending transactions
 - alert: HighPendingTransactions
        expr: bridge_pending_transactions > 50
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High number of pending transactions"
          description: "{{ $value }} transactions pending for over 5 minutes"

      # P2P connectivity issues
 - alert: LowPeerConnectivity
        expr: p2p_connected_peers < 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low P2P peer connectivity"
          description: "Only {{ $value }} peers connected"

      # Database errors
 - alert: DatabaseErrors
        expr: increase(db_errors_total[5m]) > 10
        labels:
          severity: critical
        annotations:
          summary: "Database errors detected"
          description: "{{ $value }} database errors in last 5 minutes"

      # Signature collection stuck
 - alert: SignatureCollectionStuck
        expr: bridge_signatures_collected < 3 and time() - bridge_attestation_created_timestamp > 600
        labels:
          severity: warning
        annotations:
          summary: "Signature collection taking too long"
          description: "Attestation has only {{ $value }} signatures after 10 minutes"

      # Block processing lag
 - alert: BlockProcessingLag
        expr: (bridge_chain_latest_block - bridge_latest_block_processed) > 100
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Block processing is lagging"
          description: "{{ $value }} blocks behind on {{ $labels.chain }}"

Grafana Setup

Install Grafana

# Docker
docker run -d \
 --name grafana \
 -p 3000:3000 \
 -v grafana-storage:/var/lib/grafana \
  grafana/grafana

# Default login: admin / admin

Add Prometheus Data Source

1. Go to Configuration → Data Sources
2. Add data source → Prometheus
3. URL: http://prometheus:9090
4. Save & Test

Import Dashboards

Pre-built dashboards are available at /monitoring/grafana-dashboard.json.

1. Go to Dashboards → Import
2. Upload JSON file or paste content
3. Select Prometheus data source
4. Import

Dashboard Panels

Bridge Overview

Key panels for the overview dashboard:

Per-Chain Metrics

# Header submissions per chain and path (transfer-driven vs proactive)
sum by (evm_chain, path) (rate(bridge_evm2casper_block_headers_submitted_total[5m]))

# Scanner progress per chain (never advanced by a failed scan)
bridge_evm2casper_scan_cursor_block

# Seconds since the last error-free scan
time() - bridge_evm2casper_scan_last_success_timestamp_seconds

Relayer Health

# Relayer status (up/down)
up{job="bridge-relayers"}

# Header attempts per relayer (filter relayers by instance)
increase(bridge_evm2casper_block_headers_submitted_total{instance=~"$relayer"}[$__rate_interval])

# P2P messages per relayer
rate(bridge_p2p_messages_sent_total{instance=~"$relayer"}[5m])

Filter relayers with the instance label; the dashboard's $relayer variable is populated from it. Do not sum per-relayer gauges or token-flow counters across instances: each relayer reports the same events.

Alertmanager Configuration

alertmanager.yml

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'default'
  routes:
 - match:
        severity: critical
      receiver: 'pagerduty'
 - match:
        severity: warning
      receiver: 'slack'

receivers:
 - name: 'default'
    email_configs:
 - to: 'ops@example.com'
        from: 'alerts@example.com'
        smarthost: 'smtp.example.com:587'

 - name: 'slack'
    slack_configs:
 - api_url: 'https://hooks.slack.com/services/XXX/YYY/ZZZ'
        channel: '#bridge-alerts'
        send_resolved: true

 - name: 'pagerduty'
    pagerduty_configs:
 - service_key: 'your-pagerduty-key'

Health Check Endpoint

The bridge exposes a health check endpoint:

curl http://localhost:9080/health

Response:

{
  "status": "healthy",
  "components": {
    "database": "connected",
    "ethereum": "synced",
    "casper": "synced",
    "p2p": "connected"
  },
  "uptime": 86400
}

Recommended Alerts

Critical

• Relayer down for >2 minutes
• Database connection failed
• All P2P peers disconnected
• Attestation verification failures
• SUBMISSION_PERMANENT_FAILURE in logs - an event was parked after a verified deterministic on-chain rejection (e.g. unregistered token). No relayer will retry it. Requires operator action: fix the root cause (usually an admin registration/configuration transaction), then restart relayers to un-park.
• VerificationMismatch - public Casper node execution effects contradicted a claimed event (potential forgery attempt). Backed by bridge_casper2evm_verification_mismatch_total.
• VerificationStallTimeout - an event has gone ~6h with no public-source confirmation (fabricated event or sustained public-node outage). The event is kept pending and re-checked hourly, not deleted; it recovers by itself once a public source confirms it. Backed by bridge_casper2evm_verification_stall_timeout_total (fires once per event) and bridge_casper2evm_verification_stalled_out_events (current count).

Warning

• High pending transaction queue (>50)
• Low P2P connectivity (<3 peers)
• Block processing lag (>100 blocks)
• High error rate (>5 errors/minute)

Info

• New relayer joined network
• Configuration reloaded
• Maintenance mode enabled

Stranded-Transfer Recovery Runbook

When a user's source-side action succeeded (tokens locked on EVM / burned or locked on Casper) but the destination leg fails permanently, funds are not lost - the source chain still holds them - but the transfer is stranded until an operator acts. This is the documented recovery path.

1. Detect

• SUBMISSION_PERMANENT_FAILURE alert (see Critical alerts above) identifies the parked event, its failure classification, and the decoded error name (e.g. User error: 60024 [TokenNotFound], TokenNotRegistered(address)).
• Cross-check with bridge-cli admin get-contract-status (EVM) and bridge-cli admin casper-get-status (Casper) for pause state, fee config, and registry health.

2. Classify and fix the root cause

3. Un-park and replay

For legacy attestations, parked state is in-memory. Gateway delivery records use the durable recovery policy above, not this restart procedure. For legacy attestations, after the root-cause fix lands on-chain, restart the relayers (bridge-cli stop && bridge-cli start per host, rolling). On restart the event re-enters normal processing and replays cleanly - replay protection guarantees an already-processed leg is a no-op (EventAlreadyProcessed / 60026 is the expected duplicate signal, absorbed automatically).

3b. Locks the relayer cannot see

Un-park and restart only work for events the relayer holds in state. An EVM lock the relayer never saw (source RPC pruned the block's logs; fleet rebuilt with a wiped data volume) is re-queued from the transaction hash with bridge-cli admin replay-lock --chain <id> --tx <hash> against the stopped relayer's state file, on at least RELAYER_THRESHOLD relayers. It refuses any deposit Casper has already minted. Procedure: docs/runbooks/replay-lock.md.

4. Verify closure

• Destination-side balance reflects the transfer (admin get-locked-balance / CEP-18 balance query).
• The event no longer appears in SUBMISSION_PERMANENT_FAILURE logs after two full polling cycles.
• If the transfer STILL fails deterministically after a correct fix, escalate with the full decoded error and the attestation payload - do not loop restarts.

Design note: the bridge never marks a failed submission as processed (audit-verified - cleanup only happens after confirmed success or an on-chain-confirmed duplicate). The park mechanism halts gas-burning retries of deterministically doomed payloads; it never discards them.