Building a Canonical Financial Telemetry Backend

Research Status

Current production engineering.

A quantitative model is not only its equations.

A live system must preserve what it produced, recover from failures, handle corrections, expose stable data, and support later validation.

Vyreon's backend was built around one principle:

Raw history should remain auditable while the API exposes one canonical answer.

System Responsibilities

The backend performs three main tasks:

1. Ingest telemetry messages automatically.
2. Store raw, parsed, and derived records.
3. Serve read-only data through an authenticated API.

The public architecture is:

telemetry source
    -> event inbox
    -> ingestion worker
    -> validation and parsing
    -> durable data store
    -> derived outputs
    -> read-only API
    -> authorized client

Why An Event Inbox Became An Event Stream

The live telemetry source sends compact payloads through an authenticated event inbox.

This was initially a platform workaround.

It became a useful event-source design.

Each event records:

• Which node produced the payload.
• Which market date it describes.
• Which model tag produced it.
• When it arrived.
• Its raw body.
• Its parsed payload hash.

The event inbox acts as an append-only source stream.

The database decides which event is current.

Raw And Canonical Layers

The backend separates source evidence from the current answer.

Raw Event

Every source event can be preserved.

This allows later questions such as:

• What did the live node send?
• When did it send it?
• Was the message duplicated?
• Was a date corrected later?
• Did a migration replay old dates?
• Which raw event produced the current payload?

Payload

A parsed payload represents one full telemetry emission.

It contains metadata and links to child rows.

Prior

Prior rows contain the model state issued for each horizon.

Matured

Matured rows contain realized outcomes and innovation diagnostics.

Derived Horizon

Derived rows convert excess-return state into public return and price-space fields.

Derived Volatility

Derived volatility rows summarize innovation magnitude.

Duplicate And Correction Behavior

The system must support several real cases.

Exact Duplicate

The same payload may be sent more than once.

A payload hash allows exact duplicates to be identified.

Alternate Payload For The Same Date

A date may be resent after:

• A data repair.
• A code correction.
• A missed ingestion.
• A model migration.
• A backfill.

Both versions should remain auditable.

The API should expose one current version.

Canonical Rule

The current rule is:

The latest valid received payload becomes canonical for a node and market date.

This choice is practical.

It allows the telemetry source to remain the upstream source of truth.

A later resend naturally supersedes an earlier version without deleting history.

The result is:

• Append-only raw evidence.
• Multiple valid historical versions.
• One canonical current answer.
• Repairable backfills.
• Replay-safe ingestion.

Idempotency

The ingestion worker must be safe to rerun.

If it processes the same email or payload again, it should not create inconsistent duplicate child rows.

Idempotency is enforced through:

• Message identifiers.
• Payload hashes.
• Database constraints.
• Canonical-resolution logic.

The worker can fail and restart without losing the meaning of the database.

One Writer, Many Readers

The architecture separates responsibilities.

Worker

The worker:

• Receives and validates source events.
• Parses payloads.
• Preserves lineage.
• Writes canonical records.
• Computes derived records after canonical selection.

Database

The durable data store:

• Stores durable truth.
• Enforces relationships and constraints.
• Preserves lineage.
• Answers queries.

API

The read-only API:

• Reads from the database.
• Returns stable JSON.
• Does not parse source events.
• Does not mutate model records.
• Does not contain ingestion business logic.

This separation limits failure blast radius: a source-delivery problem should not alter already canonical records, and a read-side problem should not rewrite ingestion state.

Derived Horizon Records

The raw prior contains:

• Expected excess return.
• Predictive sigma.
• Causal baseline.
• Current spot.

The derived layer computes:

mean_return
= expected_excess_return
+ baseline

ci95_return
= 1.96 * sigma_return

price_mean
= spot * (1 + mean_return)

price_low_95
= spot * (1 + mean_return - ci95_return)

price_high_95
= spot * (1 + mean_return + ci95_return)

These fields are calculated after canonical selection.

The API does not recompute them on every request.

Derived Volatility

The backend calculates innovation RMS from matured rows.

This creates a stable customer-facing volatility measurement while preserving the underlying matured observations for internal analysis.

API Boundary

The read-only API separates lineage-aware internal analysis from smaller public or customer-facing views. External consumers receive documented derived fields rather than raw records or operational implementation details.

API Authentication

Access is authenticated and scoped. The public architecture intentionally does not disclose credential formats, route definitions, secret-handling procedures, deployment topology, or operating schedules.

Why This Backend Is Research Infrastructure

The backend does more than serve a product.

It makes stronger research possible.

It Preserves What Was Issued

Strict issue-date validation requires the exact prior that existed before the outcome matured.

It Preserves Corrections

A corrected payload can become canonical without erasing the earlier record.

It Supports Reproducibility

Internal notebooks can query one consistent source rather than copy data manually.

It Separates Measurement From Presentation

The public website consumes derived output. It does not need to know how source parsing or canonical selection works.

It Enables Multi-Asset Growth

Additional nodes and symbols can use the same event, lineage, and API model.

Operational Limits

The current backend is intentionally small.

It is appropriate for:

• Daily cadence.
• A small number of assets.
• One primary operator.
• Modest API demand.

It is not designed for:

• High-frequency market data.
• Hundreds of heavy intraday jobs.
• Large public traffic.
• Multi-region failover.
• Institutional service-level guarantees.

Those capabilities should be added only when the need exists.

Security Boundary

Operational credentials, access-control configuration, deployment topology, and live endpoints are intentionally outside this public research record. The architectural point is the boundary: public documentation can explain lineage and correction behavior without exposing the controls that operate a live system.

Relational Schema

The core table relationship is:

raw_event
    -> payload
        -> prior
        -> matured
        -> derived_horizon
        -> derived_volatility

The child tables are not physically nested.

The hierarchy is expressed through foreign keys.

This allows SQL to:

• Preserve one payload with several horizon rows.
• Join a matured outcome back to its source payload.
• Query one canonical date efficiently.
• Keep raw evidence separate from derived interpretation.

Canonical Promotion Transaction

When a new valid payload arrives for an existing node and market date, canonical status changes as one atomic operation: preserve the source event, validate the new version, resolve the canonical record, and refresh the affected derived outputs. If that operation does not complete, the previous canonical answer remains intact.

Forecast-Ledger Extension

The backend can support strict issue-date validation through a dedicated ledger. Conceptually, it preserves the forecast identity, issue-time expectation and uncertainty, source version, and the later matured outcome. This separates a historical forecast record from an adaptive filter's current state without disclosing field-level production contracts.

Why A Stable API Contract Matters

Internal storage details can change as the system evolves. A stable external interface translates that implementation into a documented, durable contract rather than exposing raw records or implementation details.

Research Reproducibility Through API Access

Before the backend existed, research often required manual exports and one-off notebook state.

The API changed that workflow.

A research workflow can retrieve canonical issued states, later outcomes, and documented derived measurements from one traceable source. This reduces manual copying and lets validation, reporting, and later research begin from consistent evidence.

The backend does not guarantee that every analysis is correct.

It guarantees that the analysis can begin from one traceable data source.

Change Management

Important invariants include one canonical payload per node and date, preserved raw evidence during ordinary correction, and derived records that identify their source version and model version. These lineage guarantees are more important than exposing operational mechanics in a public research article.

Current Conclusion

The telemetry backend converts a fragile platform workaround into a durable data system.

Its strongest properties are:

• Append-only raw evidence.
• Canonical correction.
• Idempotent ingestion.
• Clear parent-child lineage.
• One writer and many readers.
• Stable derived outputs.
• Authenticated read-only access.

This infrastructure is part of the research method.

Without reliable lineage, a model cannot prove what it knew before an outcome occurred.