How to Make High-Frequency Market Data Work for Mid- and Low-Frequency Strategies

DolphinDB
2026-05-28

High-frequency market data contains a level of market microstructure detail that daily OHLC data simply can't match — order book dynamics, trade impact, informed order flow, intraday liquidity patterns. The problem is that building strategies directly on tick data is expensive: you need serious infrastructure, the signal-to-noise ratio is brutal, turnover costs eat returns, and scaling up is genuinely hard.

There's a middle path that's gaining traction in systematic trading: mid- to low-frequency factors. The idea is to extract statistically meaningful signals from minute-bar or tick-level data and compress them into stable daily features — signals that can plug directly into mid- or low-frequency strategy pipelines without requiring a tick-by-tick execution infrastructure.

This post covers how DolphinDB approached building a production-grade framework for exactly this, including 100+ pre-built factors, a parallel computation architecture, and a storage layer designed for ongoing factor management.

The Problem with Existing Approaches

Most teams doing this kind of work end up stitching something together in Python. The gap between "it works on a sample" and "it runs on years of full-market minute data" is wide. Pandas-based pipelines hit memory and speed walls fast. There's no standard template, so every researcher builds their own version, and the results don't compose well into a shared factor library.

What's been missing is a unified, standardized framework that handles the full lifecycle — computation, storage, updates, and retrieval — in a way that works for a team, not just a single researcher.

Use Cases

Signal research from high-frequency data. For quant teams that want to extract alpha from high-frequency data without going full HFT, this framework provides a ready-to-use factor pipeline. Feed in minute bars, Level-2 snapshots, or tick data; get back daily factor values ready for analysis and backtesting.

Institutional factor library management. For firms building out a research platform, this framework fits naturally into a factor asset management layer — standardized templates, batch computation, versioned storage, and incremental updates all included.

Core Features

1. Unified Factor Template

Every factor in the library follows the same interface: takes a data table as input, returns a standardized output table with SecurityID , TradeDate , Value , FactorName , and UpdateTime.

Here's an example — a volume proportion skewness factor computed from snapshot data:

def skewVolProp(snapshot){
    snap =
        select
            TradeDate, TradeTime, SecurityID,
            deltas(TotalVolumeTrade)\last(TotalVolumeTrade) as volProp
        from snapshot
        context by TradeDate, SecurityID csort TradeTime
        having TradeTime >= 09:30:00.000

    res =
        select
            SecurityID,
            TradeDate,
            skew(volProp) as Value,
            "skewVolProp" as FactorName,
            now() as UpdateTime
        from snap
        group by TradeDate, SecurityID
    return res
}

The consistent output schema means any factor can be stored, queried, and compared without format conversion. Adding a user-defined factor is straightforward — follow the same output convention and it slots into the existing management and retrieval layer automatically.

Note: Field names vary across data vendors. Align field mappings to your local data source before using the library.

2. 100+ Ready-to-Use Factors

The factor library covers four data source types, each producing daily factors through aggregation and statistical compression of high-frequency data:

Minute-bar factors — focused on intraday price-volume behavior and trading structure:

  • Shortest-path illiquidity — constructs an illiquidity measure from bar-level price paths, capturing price movement resistance
  • Volume tidal price velocity — characterizes the rhythm of intraday volume cycles and their relationship with price movement
  • T-distribution active buy proportion — uses a T-distribution to estimate the share of active buying, reflecting directional order flow bias

Level-2 snapshot factors — mining order book information and high-frequency volatility structure:

  • Realized jump volatility — separates continuous price variation from jump components to identify abnormal volatility events
  • Smart money factor — tracks passive order placement, order flow persistence, and trading asymmetry to identify informed trader activity
  • Daily bid-ask spread — measures intraday microstructure liquidity via quoted spread

Order-level factors — capturing order intent and trading expectations:

  • Post-open net buy increment ratio — tracks the ratio of net buy order changes to turnover after market open, reflecting early-session capital inflow intensity

Trade-level factors — characterizing active trading and large-order impact:

  • Large order buy proportion — identifies orders with significantly above-average volume to detect abnormal trading impact
  • Sell rebound deviation — measures the average price deviation of sell orders executed below the closing price, capturing regret-aversion behavior
  • Mega order price impact — tracks cumulative price impact of very large active trades, quantifying how outsized capital positions drive price

All factors ship with complete computation logic. Use them directly, or treat them as reference implementations for building your own.

3. Async Background Jobs + Parallel Computation

High-frequency data is large. A naive sequential approach on years of full-market minute data isn't practical. The framework addresses this at two levels.

First, computation tasks are submitted as background jobs via submitJob, freeing the client session immediately:

def factorjob(conf){
    dataTB = loadTable(conf[`dataDB], conf[`dataTB])
    days = conf[`startDay]..conf[`endDay]
    startTime = 09:30:00.000
    endTime = 14:57:00.000

    ds = sqlDS(<select SecurityID, date(DateTime) as TradeDate, time(DateTime) as TradeTime,
                       LastPx, Volume, Amount
                from dataTB
                where date(DateTime) in days
                  and (time(DateTime) between startTime and endTime)
                  and LastPx > 0>)

    diffTB = mr(ds, conf[`func]).unionAll()
    res = conf[`funcsec](diffTB)
    loadTable(conf[`factorDB], conf[`factorTB]).append!(select * from res)
}

conf = {
    func       : fuzzinessDiff,
    funcsec    : adjFuzzinessDiff,
    factorName : "fuzzinessDiff",
    dataDB     : "dfs://stockMinKSH",
    dataTB     : "stockMinKSH",
    factorDB   : "dfs://factor_day",
    factorTB   : "factor_day",
    startDay   : 2021.01.01,
    endDay     : 2021.01.31
}

id = submitJob("factorjob", conf[`factorname], factorjob, conf)

Second, within each job, DolphinDB's map-reduce framework (mr) distributes computation across data partitions in parallel, making full use of cluster resources.

Batch computation is also supported natively — upload a configured script and loop over multiple factors to compute and store them in a single run. This is well-suited for historical backfills and scheduled daily updates with minimal manual intervention.

4. Versioned Factor Storage and Incremental Updates

The framework includes a complete storage layer built on DolphinDB's distributed database, partitioned by trade date and factor name. The TSDB engine with keepDuplicates=LAST handles incremental updates automatically — re-writing a factor for an existing date/security combination retains the latest value without manual deduplication.

// Create the daily factor database
create database "dfs://factor_day"
partitioned by RANGE(date(datetimeAdd(1980.01M, 0..80*12, 'M'))), VALUE(`f1`f2),
engine='TSDB',
atomic='CHUNK'

// Create the factor table
create table "dfs://factor_day"."factor_day"(
    SecurityID  SYMBOL,
    TradeDate   DATE[comment="time column", compress="delta"],
    Value       DOUBLE,
    FactorName  SYMBOL,
    UpdateTime  TIMESTAMP,
)
partitioned by TradeDate, FactorName
sortColumns=[`SecurityID, `TradeDate],
keepDuplicates=LAST,
sortKeyMappingFunction=[hashBucket{, 500}]

Computing a factor once and throwing the result away is a research exercise. Building a factor library means storing, versioning, and continuously updating — that's what turns a calculation into a reusable asset.

5. Wide and Narrow Table Conversion in One Query

Factor data stored in narrow (long) format is efficient for updates and maintenance. But cross-sectional analysis — comparing multiple factors across securities on a given date — is often easier in wide format.

DolphinDB's pivot by handles this in a single statement:

dailyFactor = loadTable("dfs://factor_day", "factor_day")

factorTB1 = select
                Value
            from dailyFactor
            where FactorName in `skewVolProp`netBuyIntenOpen
            pivot by SecurityID, TradeDate, FactorName

This produces a wide feature matrix across selected factors — ready for correlation analysis, factor screening, or use as model input features.

Wrapping Up

High-frequency data is rich with market microstructure information, but it only translates into alpha when you can extract it reliably and at scale. The DolphinDB mid- to low-frequency factors framework is built around that challenge: 100+ factors derived from minute bars, Level-2 snapshots, and tick data, with a full pipeline covering computation, storage, updates, and retrieval.

The goal isn't just to compute factors — it's to turn high-frequency data into a sustainable, queryable, reusable factor asset library that a team can build on over time.

For the full factor list, detailed computation logic, and setup instructions, see: https://docs.dolphindb.com/en/Tutorials/hf_to_lf_factor.html