Data Monitoring

Overview

The data monitoring module is responsible for acquiring, normalizing, and storing market data that feeds into the analysis engine.

Requirements

• Latency: Near real-time (< 1 minute delay)
• Coverage: Forex major pairs + selected stocks/indices
• Granularity: 1-minute candles, tick data where available
• History: Minimum 30 days rolling window for backtesting
• Reliability: Automatic reconnection, data gap detection

Watched Instruments

Forex Pairs (Initial)

Pair	Session	Notes
EUR/USD	London + NY	Most liquid, tight spreads
GBP/USD	London + NY	High volatility windows
USD/JPY	Tokyo + London	Trend-following opportunities
AUD/USD	Sydney + London	Commodity correlation

Indices (Future)

Index	Session	Notes
S&P 500 (SPX)	NY	Broad market sentiment
NASDAQ 100 (NDX)	NY	Tech sector proxy

Data Pipeline

┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  API Source  │────▶│  Normalizer  │────▶│   Storage    │
│              │     │              │     │              │
│ • REST poll  │     │ • OHLCV fmt  │     │ • SQLite/TS  │
│ • WebSocket  │     │ • Timestamps │     │ • Partitioned│
│ • Fallback   │     │ • Validation │     │ • Indexed    │
└──────────────┘     └──────────────┘     └──────────────┘

Data Schema

CREATE TABLE candles (
    id INTEGER PRIMARY KEY,
    symbol TEXT NOT NULL,
    timeframe TEXT NOT NULL,  -- '1m', '5m', '15m', '1h'
    timestamp DATETIME NOT NULL,
    open REAL NOT NULL,
    high REAL NOT NULL,
    low REAL NOT NULL,
    close REAL NOT NULL,
    volume REAL,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(symbol, timeframe, timestamp)
);

CREATE INDEX idx_candles_symbol_time ON candles(symbol, timeframe, timestamp DESC);

Health Checks

• Data freshness monitoring (alert if > 5 min stale)
• Gap detection and backfill
• Source failover (primary → secondary API)
• Rate limit tracking per API key