How to Build a Real-Time Trade-Order Matching Pipeline

Every trading system continuously produces two independent streams of market events: orders and trades. A trade tells you what happened — symbol, price, quantity. It never tells you why. Which buy order and which sell order actually matched to produce it?

If you run transaction cost analysis, market-making analysis, or trading-behavior monitoring, that missing link isn't a footnote — it's the whole analysis. Once a trade is tied back to its originating orders, you can measure how long an order sat in the book before it filled, compare the fill price against the order's original price, and separate passive fills from aggressive ones. String that together across an order's life and you've reconstructed its full lifecycle, live.

The catch: orders and trades arrive as two independent, unsynchronized streams. Wait until both sides are persisted to disk before matching them, and "real-time monitoring" quietly becomes "monitoring with a lag nobody signed up for."

This is precisely the problem DolphinDB's Streaming Left Semi Join Engine was built to solve.

The Shape of the Solution

A left semi join engine continuously pairs records from two streaming tables on a shared key. Here, trades are the primary stream; orders are the reference stream that enriches each trade when a matching order becomes available.

But every trade points to two orders — a buyer and a seller — so one join isn't enough. The pattern is to chain two left semi join engines:

The first engine matches each trade against its sell order.
The second engine takes that partially-enriched trade and matches it against the buy order.

A trade only reaches the final output once both lookups succeed — no batch window, no polling, just enrichment as data arrives.

Building the Pipeline

In this example, each trade carries two references: BuyNo and SellNo, pointing to the buy order and sell order that produced it. Since both order references must be resolved before the trade is complete, we build the pipeline by chaining two Left Semi Join engines.

Creating the Streaming Join Engines

Start by creating streaming tables.

share(table=streamTable(1:0, `Sym`BuyNo`SellNo`TradePrice`TradeQty`TradeTime,
[SYMBOL, LONG, LONG, DOUBLE, LONG, TIME]), sharedName=`trades)

share(table=streamTable(1:0, `Sym`OrderNo`Side`OrderQty`OrderPrice`OrderTime,
[SYMBOL, LONG, INT, LONG, DOUBLE, TIME]), sharedName=`orders)

The trades table contains the trade information together with the buy and sell order IDs (BuyNo and SellNo), while the orders table stores the corresponding order details.

Now create the two Left Semi Join engines:

// match the buy-side order
ljEngineBuy = createLeftSemiJoinEngine(
    name="leftJoinBuy", leftTable=outputTemp, rightTable=orders, outputTable=output,
    metrics=<[SellNo, TradePrice, TradeQty, TradeTime, BuyOrderQty, BuyOrderPrice, BuyOrderTime,
              OrderQty, OrderPrice, OrderTime]>,
    matchingColumn=[`Sym`BuyNo, `Sym`OrderNo])
    
// match the sell-side order
ljEngineSell = createLeftSemiJoinEngine(
    name="leftJoinSell", leftTable=trades, rightTable=orders, outputTable=getLeftStream(ljEngineBuy),
    metrics=<[BuyNo, TradePrice, TradeQty, TradeTime, OrderQty, OrderPrice, OrderTime]>,
    matchingColumn=[`Sym`SellNo, `Sym`OrderNo])

The parameter to focus on is matchingColumn. For the sell-side engine, a trade's SellNo is compared against an order's OrderNo (with Sym required to match too, since order numbers reset per symbol). The buy-side engine does the same thing with BuyNo.

Wiring the Streams Together

Finally, subscribe each stream to the corresponding side of the join engines.

subscribeTable(tableName="trades", actionName="appendLeftStream",
    handler=getLeftStream(ljEngineSell), msgAsTable=true, offset=-1)
subscribeTable(tableName="orders", actionName="appendRightStreamForSell",
    handler=getRightStream(ljEngineSell), msgAsTable=true, offset=-1)
subscribeTable(tableName="orders", actionName="appendRightStreamForBuy",
    handler=getRightStream(ljEngineBuy), msgAsTable=true, offset=-1)

Wiring the Streams Together

Finally, subscribe each stream to the corresponding side of the join engines.

subscribeTable(tableName="trades", actionName="appendLeftStream",
    handler=getLeftStream(ljEngineSell), msgAsTable=true, offset=-1)
subscribeTable(tableName="orders", actionName="appendRightStreamForSell",
    handler=getRightStream(ljEngineSell), msgAsTable=true, offset=-1)
subscribeTable(tableName="orders", actionName="appendRightStreamForBuy",
    handler=getRightStream(ljEngineBuy), msgAsTable=true, offset=-1)

Once the subscriptions are established, the matching process runs automatically as new streaming data arrives.

Seeing It Work

Feed in a handful of orders, then a few trades that reference them:

// generate data: trade
t1 = table(`A`B`B`A as Sym, [2, 5, 5, 6] as BuyNo, [4, 1, 3, 4] as SellNo, [7.6, 3.5, 3.5, 7.6]as TradePrice, [10, 100, 20, 50]as TradeQty, 10:00:00.000+(400 500 500 600) as TradeTime)
// generate data: order
t2 = table(`B`A`B`A`B`A as Sym, 1..6 as OrderNo, [2, 1, 2, 2, 1, 1] as Side, [100, 10, 20, 100, 350, 50] as OrderQty, [7.6, 3.5, 7.6, 3.5, 7.6, 3.5] as OrderPrice, 10:00:00.000+(1..6)*100 as OrderTime)
// input data
orders.append!(t2)
trades.append!(t1)

The correspondence of records between the input streams is shown below:

Input Data Correspondence

The output table fills in automatically: each trade now carries the quantity, price, and time from both its buy-side and sell-side orders — everything you need for downstream TCA or lifecycle analysis, generated the instant the last piece of the puzzle arrives.

Output Table

Why This Pattern Is Worth Knowing

The trick generalizes past trading. Any time an event only becomes meaningful after it's correlated with context sitting in a second stream — a payment matched to an invoice, a support ticket matched to a customer record, a sensor reading matched to a maintenance log — chaining left semi join engines gets you there without batching, polling, or a round trip through storage.

For quant infrastructure specifically, this means TCA, market-making analysis, and behavior monitoring can all run on live data instead of end-of-day reconciliations. The matching logic lives entirely inside the streaming pipeline; everything downstream just consumes fully-enriched records as they show up.