Market Data Management
Market Data Ingestion
Connect to exchanges, broker channels, and third-party vendors from a single platform — compatible with direct exchange feeds, institutional market data services, and leading providers including ICE, Refinitiv, and Bloomberg.
Bulk import of historical data is supported in CSV, HDF5, Parquet, and other formats, with dedicated modules for Level-2 data, and migration plugins for MySQL, KDB+, InfluxDB, and other systems.
Featured Tutorials & Best Practices:
Market Data Storage & Querying
Full-market tick data is stored in a distributed time-series database, partitioned by date, security, and market. Columnar storage and high-compression algorithms support TB-to-PB-scale data at 4:1–10:1 compression ratios, cutting storage costs by over 70%. A distributed SQL engine delivers second-level full scans and millisecond-level slice access, with in-database support for windowing operations, factor computation, and order book feature extraction.
Featured Tutorials & Best Practices:
- Multi-Model Database
- Data Partitioning
- Best Practices for Financial Data Storage
- Practical Approaches to Storing Multi-Level Market Snapshots
- Distributed SQL Query
Market Data Integration & Cleansing
Consolidates trade executions, order entries, and snapshots across equities, futures, options, and ETFs into a unified and normalized data feed. The real-time cleansing pipeline handles multi-source alignment, deduplication, out-of-order correction, gap filling, and anomaly removal — reconstructing order books and multi-level snapshots from tick data. The same rules apply to historical imports for consistent and quality-controlled output.
OHLC Bar Generation
Generates multi-period candlesticks in real time or in batch mode from tick or snapshot data across equities, funds, futures, and options. Supports OHLC calculation for any timeframe or custom interval, with additional support for continuous contracts and synthetic indices.
Featured Tutorials & Best Practices:
Real-Time Indicator Computation
Market events are ingested and processed end-to-end in microseconds by the built-in streaming engine — delivering real-time computation of returns, drawdown, positions, turnover, slippage, and factor exposure. Time-series, cross-sectional, and multi-source engines support incremental computation with minimal overhead, and most indicators can be built in just a few lines of script with rich operators.
Featured Tutorials & Best Practices:
Market Data Replay
Replays historical tick and order book data in original timestamp order to faithfully recreate the live market environment. Replayed data is injected into stream tables, allowing backtesting frameworks and strategy platforms to operate as if connected to a live feed. Supports single table (1-to-1), and multi-table (N-to-N, N-to-1) replay modes at real-time or accelerated speed.
Featured Tutorials & Best Practices:
Distribution & Visualization
Distributes standardized data to all downstream systems via publish/subscribe — trading, risk, research, and monitoring dashboards all drawing from a single, consistent data stream. Custom metrics can be defined rapidly and output in real time, with built-in visualization components and third-party BI tool support for agile dashboard and research interface development.
Featured Tutorials & Best Practices:
- Stream Subscription: Intra-node Subscription, Inter-node Subscription within a Cluster, Inter-node Subscription Across Clusters,