API ingestion recovery

The API indexes bridge activity independently of relayer execution. A completed on-chain transfer can have delayed API visibility while an ingestion obligation is unresolved. /livez measures process liveness; stream loss does not trigger process restarts.

Code ownership

The API pins the private npm package @astralbeam-io/protocol@0.1.0 for integer/address decoding and dictionary hashing. Its local adapter preserves existing event acceptance and owns factory attribution, safe-number limits and recovery policy. This dependency change does not alter API responses or require a database migration. API builds require authorized private npm read access; missing access is a build failure, not a reason to substitute local source.

The relayer keeps its own CES parser, factory resolver, manifest and build dependencies. The API-local adapter is not a relayer dependency. The relayer independently pins @astralbeam-io/protocol@0.2.0 for fixed-width integers, dictionary hashes, deposit IDs and the exposure-control wire formats and storage layouts, retaining its own event acceptance policy. Relayer builds also require private npm read access. Indexer changes do not implicitly upgrade signing-evidence decoding. The relayer retains independent verification, finality/provider policy, custody, signing and submission; API completion is not signing authority. An authorized rollback uses the preceding reviewed API image without deleting recovery data or changing the relayer.

Recovery model

Casper REST polling obtains transaction execution effects from CASPER_NODE_URL. The API-local @astralbeam/casper-events package resolves the configured TokenFactory package, attributes its CES dictionary writes, authenticates dictionary addresses and decodes TokensBurned, NativeTokenLocked, TokensMinted and NativeTokenUnlocked. Other token-contract events in the same execution are excluded. Submitted arguments never establish a transfer or its nonce. Source block/time comes from the executed transaction's RPC block, not recovery wall-clock time.

CASPER_NODE_URL is mandatory when Casper indexing is enabled, including combined API startup. Durable RPC recovery runs without a CSPR.cloud REST key; discovery polling still requires that key. In recovery-only mode, a completed recovery poll updates the poll-success timestamp; a thrown recovery failure leaves it unchanged. This is a degraded recovery capability, not a healthy replacement for full ingestion: new REST discovery stops. Recovery-only configuration returns degraded HTTP 503 on both fresh and stale databases; expected configured chains require independent recent discovery evidence. See API readiness. Do not refresh those timestamps solely to make readiness green.

Before advancing a cursor past a failed handler, the API stores an ingestion_recovery obligation. An unsuccessful queue write holds the cursor. Retry identity is chain/contract scoped; processing failures retain their original age and retry on capped backoff. Expected finality/source waiting uses a short retry cadence without incrementing the failure counter. Finalized EVM burn arrival wakes matching chain/nonce unlock obligations. EVM retries re-read canonical logs and respect the finalized ceiling. They do not make provisional destination observations final.

Casper mint completions use the existing pending_mint_events table. Native unlock completions use durable recovery storage. Both survive process restart and retry when the source is indexed. The mint queue rotates attempted entries so old unmatched items do not monopolize a bounded batch.

Provisional transaction tracking

Casper burn and native-lock WebSocket notifications create a transaction immediately with sourceVerified: false. Source block and timestamp remain unknown until independent RPC verification. RPC replaces provisional source fields and marks the same row verified atomically; it does not trust a populated WS hint as proof.

An early EVM Unlocked or WrappedMinted notification also creates or updates that same transaction row. Its block/log-bound observations and removal tombstones are stored in row metadata, not a separate destination table. Source arrival and periodic EVM polling reconcile those observations against the configured chain/locker, token, transfer type, recipient and unchanged raw amount. Conflicting hints are not arbitrarily selected.

A matching destination observation shows submitted. Completion requires both verified Casper execution and canonical finalized EVM evidence. Canonical destination evidence and its non-completed recovery owner survive restart and cursor advancement until exact source correlation succeeds or stable finalized replacement evidence invalidates the destination, not merely until source verification finishes. Missing mappings, invalidated sources, payload/domain disagreement and ambiguous canonical destinations remain waiting with full evidence; source-first refusals retain their existing parked/cursor policy. Mapping repair can retry automatically; contradictions and verified sources lacking a token identity require separately approved source/metadata reconciliation, never inference from the destination. Removed or orphaned hints are cleared; late duplicates cannot resurrect a removed event. Confirmed statistics exclude unverified sources, while transaction tracking remains available.

A stable finalized replacement block that omits a waiting destination event atomically invalidates its recovery owner and its exact chain/locker/transaction/log/block observation. That observation remains an explicit invalidated tombstone, excluded from reconciliation and retained across restart and pruning. Later Casper source proof cannot complete from it; WS duplicates cannot restore it. If Casper correlation completed the transaction before the still-active EVM owner retries, invalidation also removes that exact destination projection and its satisfied signature count, retaining the verified source as finalized. Both source/invalidation orderings serialize under the transaction row lock. Already-terminal recovery owners are not reopened by invalidation. Pruning retains canonical observations until both the exact EVM recovery owner and matching transaction projection are completed; source correlation alone cannot discard the identity needed by an active owner to revoke that projection. Independently verified finalized reappearance may restore eligibility for the proven inclusion. New-block reappearance retains the old tombstone as audit evidence. This does not periodically revalidate already-completed owners.

Source retries reuse ingestion_recovery. This does not change relayer verification, recover previously lost WS notifications, or guarantee delivery during a database outage. Canonical polling remains the fallback.

Do not roll back readers or indexers to binaries without the source-verification gate while unverified rows exist (SELECT COUNT(*) FROM transactions WHERE source_verified = FALSE). Stop ingestion and retain a compatible reader; use a forward fix or separately reviewed recovery plan. Preserve the additive columns and pending evidence.

Correlation boundaries

All four transfer families require the same chain-scoped token mapping, recipient and raw amount on both legs before completion or destination-first source enrichment. The pinned contracts forward the signed raw U256 unchanged; display decimals do not authorize rescaling. Missing mappings and individual payload mismatches preserve the original row and durable evidence. Mint mismatches retain their original event JSON and a separate pending_mint_events.correlation_error; other mismatches retain parked or destination-correlation waiting recovery evidence.

The current Casper source transaction representation assumes one source transfer per transaction. Multiple source events and ambiguous hash-only lookups remain unresolved. Independent authenticated event categories still run; a deploy parks after retryable siblings have succeeded, retaining every blocked event. Event-scoped source and destination correlation is required before such transactions can materialize completely.

NativeTokenUnlocked does not emit a source chain. After successful factory-attributed execution, a hash-validated direct call to the configured factory package supplies the signature-bound source chain from unlock_native or unlock_native_with_signatures. The chain and available RPC destination block/time are persisted for restart-safe, chain-scoped nonce correlation. SDK decoding validates the returned transaction hash and approvals, not execution/finality by itself; those are trusted-node evidence. Captured public-node unlock JSON, including approvals, is covered by regression tests. Proxy/session arguments cannot establish the inner call context. Successful unlocks whose validated transaction cannot bind a direct factory source chain are parked as unsupported_unlock_context; multiple unlock events are parked as multiple_unlock_events, with all canonical unlocks retained as deploy-level evidence. Missing or malformed unlock transaction JSON remains retryable. The block is read and validated before unlock-only context rejection; independent source events can use that same validated execution/block evidence. These evidence errors and RPC failures increment the processing-failure counter and retry on capped exponential backoff without expiring the obligation. Authenticated unlocks awaiting a late source remain deferred on the short cadence. REST discovery parks proven unsupported execution contexts immediately after durable enqueue, retaining any canonical evidence before cursor advancement; a failed parking write holds the cursor. No path completes against an unscoped nonce, even when only one matching row is currently indexed. Transaction-index correlation for Casper mints remains distinct from EVM log-index retry identity. Migration 022 adds an actual chain-scoped receipt log key for EVM lock/burn rows, while retaining transaction position separately. Casper mint completion also requires a nonce recovered from a successful, hash-validated direct factory invocation; ambiguous or unavailable context remains pending. See event identity migration and recovery. This is single-process recovery, not distributed writer fencing; preserve the single-replica/Recreate deployment constraint pending #105.

Unsupported execution-result shapes, including legacy Version1 execution effects, are parked instead of interpreted as successful emptiness. Version2 failed execution emits no transfers. Factory-attributed malformed transfer events fail closed; a malformed FeeCollected is ignored only when its dictionary write and raw event-name prefix independently authenticate it as unrelated. Same-chain nonce reuse across locker redeployments remains ambiguous and is never guessed.

Parked rows retain payload, age, failure class (parked:*) and explicit state; they are excluded from automatic retries and active-work age. They remain included in total pending and are separately exposed by bridge_api_recovery_parked. Inspect and alert on this gauge; a zero active-age metric does not mean all obligations are resolved. Explicit approved historical replay reactivates validated work; duplicate stream delivery does not unpark it.

Bounded historical reconciliation

A cursor already advanced past an incident does not rewind when the application is upgraded. Preview an explicit set of 1-100 exact transaction hashes:

npm run build --prefix api
npm run backfill:source --prefix api -- --dry-run <64-hex-transaction-hash>

Set CASPER_TOKEN_FACTORY_HASH and CASPER_NODE_URL for the intended environment. Preview fetches execution and block evidence and prints authenticated event data. It neither opens the database nor moves a cursor. Captured source and unlock receipts are recorded in api/tests/fixtures/recovery-*.json; the capture script performs only public-chain reads, retains node transaction JSON, and accepts explicit transaction hashes plus the same factory/RPC environment settings.

After separate execution approval, use the same hashes with --execute --confirm. Execution enqueues idempotent obligations and explicitly reactivates matching parked deploy/unlock obligations for the normal indexer; it does not force transaction completion or rewind shared cursors. A partially interrupted queue batch can be repeated safely. Preview is not proof that every correlation will resolve; inspect the boundaries above and queue age afterward. Do not use the older destination-only backfill script to recover missing source events.

Rollout prerequisites

Configure a valid CSPR_CLOUD_ACCESS_KEY for normal full-ingestion rollout. Recovery-only mode without the key is degraded and does not establish readiness. Apply general migrations through 028_provisional_transactions.sql (including source canonicality026 and public keys027) before starting the new indexer image; #108 migration 022_evm_event_identity.sql is the required base. Verify the recovery tables and the transactions.source_verified and transactions.destination_observations columns exist in the target database before rollout; missing recovery storage intentionally pauses ingestion. Preserve single-replica/Recreate deployment. Render the target chart and verify a container port named metrics resolves to 9091, then verify the real ServiceMonitor target and a successful scrape during the approved rollout. A local render does not prove a live scrape.

Metrics listener

API_METRICS_PORT enables a separate /metrics listener. The default bind address is 127.0.0.1; Kubernetes values explicitly set API_METRICS_HOST=0.0.0.0 and port 9091, exposed by a ClusterIP service and ServiceMonitor, not the public HTTPRoute. Configure network access to that operational port through cluster policy. The standalone indexer starts this listener. The combined entry point starts it before checking RUN_INDEXERS, including API-only mode and an empty EVM chain configuration. Metrics availability alone does not establish that indexers are running; monitoring must check the configured ingestion/readiness signals.

Metrics use bounded configured chain labels, never provider URLs or transaction hashes:

• bridge_api_stream_configured, bridge_api_stream_connected: distinguish optional polling-only EVM chains from degraded configured streams.
• bridge_api_stream_reconnect_total, bridge_api_stream_last_connected_timestamp_seconds, bridge_api_stream_disconnected_seconds: connection history independent of polling.
• bridge_api_poll_last_success_timestamp_seconds: successful recovery polling; does not imply the queue is empty.
• bridge_api_head_scan_last_success_timestamp_seconds: EVM pre-finality visibility, separate from finalized-cursor health. Empty successful scans count as success.
• bridge_api_recovery_pending, bridge_api_recovery_parked, bridge_api_recovery_oldest_age_seconds: total unresolved work (retryable, waiting, parked), parked subset and oldest active work respectively; Casper includes pending mints and unlocks.
• bridge_api_telemetry_available: failed queue-stat reads explicitly report unavailable telemetry rather than healthy zeroes.

Alert runbooks

Authoritative rules live in make-infra charts/alerts/templates/astralbeam-api.yaml. Helm-rendered promtool tests exercise firing and resolution. The service-repo mirror is not deployment authority. Rule tests do not verify live scrape discovery, Alertmanager silences/inhibition or Slack delivery; verify those independently during an approved rollout.

AstralBeamApiStreamDisconnected

A configured stream stays disconnected for five minutes. Check stream connection and reconnect metrics, provider status and API connection logs. Compare HTTP polling and recovery age independently. Quiet contracts and intentionally polling-only chains are not stream failures. CSPR.cloud application-level Ping frames satisfy the Casper liveness deadline; protocol pong is also accepted. A silent current socket is retired, and stale sockets cannot clear the current deadline. The listener retries indefinitely with capped backoff; restarting is not a substitute for restoring the upstream connection.

AstralBeamApiRecoveryOverdue

An unresolved event is older than five minutes for a further five-minute hold. Inspect its scoped recovery row, kind, failure class and retry history. Verify the actual transaction execution and token metadata. Review mint/unlock attribution and arrival order before concluding relayers failed.

AstralBeamApiRecoveryStalled

An unresolved event is older than fifteen minutes for a further five-minute hold. Use the bounded read-only preview to inspect authenticated execution. Resolve missing metadata, provider access or correlation evidence before approving historical queue writes. Do not delete obligations or mark rows completed merely to clear the alert.

AstralBeamApiPollStalled

Successful HTTP recovery polling is stale for five minutes with a further five-minute hold. A fresh poll timestamp means the loop ran, not that every recovery item succeeded or the RPC provider is healthy. Per-item failures can coexist with a fresh timestamp; inspect recovery pending-work and oldest-active-age metrics for unresolved work. Check API RPC/REST availability, credentials, database connectivity and held pagination state. A healthy stream does not replace historical recovery polling.

AstralBeamApiHeadScanStalled

EVM head scanning is stale even if finalized polling remains healthy. Check eth_getLogs availability and range limits. Legitimate confirmation waiting does not itself invalidate successful head scans; do not reduce finality requirements to clear an ingestion alert.

AstralBeamApiTelemetryMissing

An operational scrape is down, a known API service lacks a metrics scrape, a metric read reports unavailable, or a successful scrape lacks stream instrumentation. Check the metrics port, ServiceMonitor, loaded target labels and database metrics reads. Dev service identity is astralbeam-api; public-testnet identity is astralbeam-api-testnet, in their respective make-astralbeam-* namespaces. Absence of data is not a healthy state.

Audited recovery lifecycle (#117)

ingestion_recovery keeps evidence after resolution. No TTL or retry count deletes unresolved work. history appends the previous/new state, reason, timestamp and supporting evidence atomically with each state transition; revision changes with lifecycle/context writes. Original transport payloads remain in provenance. The supported topology remains one indexer writer per database; HTTP-only processes use RUN_INDEXERS=false. This does not implement distributed ownership or fencing and does not close #105.

Null blocks, RPC errors, unchanged blocks with missing logs, inconsistent log/block attribution, and changing canonical responses cannot invalidate evidence. An invalidated original identity can transition explicitly back to retryable when finalized discovery proves that same chain/locker/transaction/log identity in a canonical block, preserving its old block payload in history before retrying the handler. It moves to completed only after handler success. A changed log index has its own #108 identity and recovery key. This does not rewrite destination history or infer completion for a different locker.

For Locked and WrappedBurned, queue invalidation and transactions.source_invalidated = TRUE commit in one SQL transaction. The stored source payload and destination history remain intact, but invalidated sources are excluded from pending/correlation lookups, transaction endpoints and newly computed statistics. The aggregate /stats endpoint retains its existing 30-second cache. Verified finalized reappearance checks stable block identity and the exact canonical log, then restores the same row at the new block position; interrupted restoration remains retryable. Other source identities remain untouched.

bridge_api_recovery_invalidated counts canonical invalidations separately. bridge_api_recovery_pending excludes completed, invalidated and operator-resolved states; it includes parked work. A zero active-age gauge is compatible with parked obligations and needs that separate gauge for interpretation. Casper pending also includes the existing pending-mint table. Bookkeeping failure logs are operational errors even when unrelated events progress; investigate them rather than treating a moving unrelated cursor as queue health.

Recovery and pending-mint schedulers visit at most 100 rows per pass using a finite ID cycle. Each captures an active-row high-water ID at cycle start; later arrivals cannot extend that cycle. Selection advances before handler/bookkeeping writes, and never refills from a new cycle in the same pass (no same-batch duplicates). A cycle containing N active rows ends within floor(N/100)+1 passes, even if rows become due after their position was passed. Such a row enters the next cycle: if L active rows have IDs at or below it, service takes at most another floor(L/100)+1 passes. Continuous higher-ID arrivals cannot increase L. Bounds assume selection queries and poll calls keep running; backoff still controls eligibility. Restart preserves every durable obligation and starts a fresh finite cycle, not a required starvation escape hatch. A failed enqueue/park in discovery holds block and pagination progress across the affected batch while independently durable siblings continue. Finalized EVM destination failures retain the full log in durable recovery. Payload gaps also retain parked evidence. Destination failures, including correlation refusals and failed enqueue/park writes, preserve the existing policy of holding the range cursor after running siblings.

REST and WS share one Casper deploy owner. Repeated parked/completed delivery consults that owner before RPC or event parsing. A mismatching WS payload cannot complete against different RPC event data; normal replay independently reads authenticated execution. Migration preserves existing dual-key rows as aliases (canonical_id) and carries their old payload/history into owner provenance; aliases are excluded from due selection and metrics. Migration does not choose an arbitrary source or delete old parked records. If either REST/WS branch is retryable or waiting, its canonical owner stays active; only two parked branches coalesce into a parked owner. At runtime, transient siblings take retry precedence, then waiting siblings, then parking. A blocked sibling’s evidence remains in transition history while a waiting unlock can still be woken by its source arrival. After all actionable siblings succeed, remaining blocked work parks rather than being marked completed.

Casper terminal classifications retain the execution variant and SHA-256 of the JSON execution result, transaction hash and JSON digest, canonical block hash/height/time, and decoded events. Event JSON over 64 KiB is represented by count and digest instead. Digests commit to the exact trusted-node response components; they are not independent consensus or RPC-signature proofs. Unsupported/ambiguous calls and zero-event/failed execution retirement require valid transaction JSON/hash/approvals and coherent canonical block metadata. Missing or malformed proof stays retryable. Single supported source events continue to use factory-bound execution effects, not outer transaction arguments; any terminal source-conflict parking additionally validates the transaction. Per-transition evidence survives duplicates and restart.

Preview-only operator resolution

Use a database account restricted to SELECT for this tool. Run from api/; it imports classes without starting a server or indexer. All commands here are read-only:

npm run recovery:preview -- --snapshot OBLIGATION_ID ROW_ID legacy_evm_chain
npm run recovery:preview -- --request reviewed-request.json

The other supported modes are legacy_casper_chain and locker_nonce_reuse. The snapshot command returns complete obligation/candidate rows and a SHA-256 snapshotFingerprint. The request JSON is strict: mode, obligationId, rowId, chain (known chain name), locker (exact address), transactionHash (exact EVM receipt hash), logIndex, and that fingerprint. Configure RECOVERY_EVM_RPC_URL for the independently verified chain. Casper legacy previews also require CASPER_NODE_URL and RECOVERY_CASPER_FACTORY matching the obligation scope.

A positive EVM legacy preview binds a null-chain source row to its exact chain, locker, successful finalized receipt and log index, including token, sender, recipient, amount, nonce and block/transaction position. A Casper legacy source_conflict preview additionally verifies the unique factory source event and destination receipt's Casper transaction key, token, amount and recipient. TokensBurned accepts only burn/unlock rows with Unlocked; NativeTokenLocked accepts only lock_native/mint_wrapped rows with WrappedMinted. Cross-family rows are refused before any candidate or proposed transition. Contradictory, missing, unknown-chain or stale evidence is refused. A redeployment preview can identify an exact source candidate; it leaves destination completion unresolved because a source receipt alone does not prove which locker an ambiguous unlock refers to. Its proposedTransition is null; it cannot terminate that unresolved obligation.

A verified legacy mapping proposes an operator_resolved identity transition, conditional on successful ordinary handler replay after applying the verified mapping with executionAuthorized: false; this is a review artifact, not an applied resolution. There is no --execute, --confirm, apply endpoint or resolution executor. Separate execution authorization and tooling would need to revalidate the complete snapshot and canonical evidence, preserve terminal transaction fields, and atomically append the resolution history with operator, approvalRef, reason, snapshotFingerprint and evidenceFingerprint. The schema rejects an operator-resolved state without these audit fields. The existing historical source-backfill tool has its own separate authorization boundary; it is not a completion or operator-resolution tool.

Migration and rollback stop checks

Stop the single indexer before applying migrations; preserve database backup and migration receipts under the deployment's separately approved procedure. Migration 023 is additive and restartable, retains old payloads/age/parked states, creates canonical deploy owners where needed and aliases old unlock keys. Migrations 025 and 026 add mint mismatch evidence and the source canonicality flag. Check owner/alias counts, parked reasons, invalidated transaction counts and state totals before resuming. Missing lifecycle schema deliberately fails ingestion rather than allowing cursors past unsaved work.

An older binary ignores the new state, alias and source canonicality fields. Older API readers also expose invalidated source projections, so keep a compatible reader during rollback. Do not resume an older indexer when any alias, canonical-invalidated, completed or operator-resolved row exists; it can retry historical evidence incorrectly. Stop first and inspect:

SELECT state, COUNT(*) FROM ingestion_recovery WHERE canonical_id IS NULL GROUP BY state;
SELECT COUNT(*) AS aliases FROM ingestion_recovery WHERE canonical_id IS NOT NULL;
SELECT COUNT(*) AS invalidated_sources FROM transactions WHERE source_invalidated = TRUE;
SELECT COUNT(*) AS incompatible_with_old_reader FROM ingestion_recovery
WHERE canonical_id IS NOT NULL OR state IN ('completed','canonical_invalidated','operator_resolved');

Keep the additive schema and evidence intact during application rollback. A forward fix or separately reviewed compatibility plan is required when the stop check is nonzero. Do not drop the table, erase history, unpark all rows, rewind shared cursors or execute historical repair as a rollback shortcut. #108 identity rollback checks still apply independently. No live migration, repair, deployment or rollback was exercised by the local #117 fixtures.