Behind the Build: One Developer's Five-Year Journey to Bridge Two Industry-Leading Engines
As a practitioner who has worked extensively with both DolphinDB and DuckDB, Boye has a clear-eyed view of what each engine does best.
DolphinDB excels at unified stream-batch processing, native distributed architecture, and best-in-class time-series performance — the right tool for large-scale real-time computation and production workloads. DuckDB shines at local embedded analytics, powerful complex SQL support, and direct file querying — ideal for lightweight, fast exploration.
Two great engines. One glaring gap.
Five Years of Workarounds
For five years, Boye worked deep in financial market data — ingesting, cleaning, aggregating, and analyzing tick-level data with DolphinDB at the core. He knew its strengths intimately, and he also knew where it handed off: complex nested SQL, local file reads, rapid ad-hoc exploration. That's where DuckDB came in.
But getting data between the two systems was a grind. Every workflow required Python glue code or CSV roundtrips, and the costs piled up fast:
- Python running out of memory on large datasets
- Slow CSV imports and exports eating into analysis time
- Parquet and JSON files requiring format conversion before ingestion
- Nanosecond timestamp precision getting silently truncated
- Type mismatches surfacing unpredictably and corrupting results
- Analysis results needing to be exported and re-imported repeatedly, dragging out quantitative research cycles
"Every day was a fight against inefficient data shuffling. Both engines are world-class — but a wall stood between them."
That wall had to come down.
The Plugin He Wished Existed
Boye set out to build a lightweight, stable, high-performance DolphinDB plugin that would make the two engines genuinely interoperable — not through workarounds, but natively.
The result is the DolphinDB DuckDB Plugin , now live on the DolphinDB Plugin Marketplace.
What it solves
No more glue code — native two-way connectivity. Through simple connect, query, and append interfaces, you can read from and write to DuckDB directly inside a DolphinDB script. No Python layer, no intermediate files. One line of code moves data in either direction.
SQL dialect complementarity — reuse your existing scripts. Delegate complex nested window functions, recursive CTEs, and multi-level subqueries to DuckDB as an "external operator," freeing up DolphinDB's core resources and keeping clusters stable. Existing DuckDB or PostgreSQL analysis scripts require no rewriting — they run as-is, with live access to DolphinDB data.
Direct file querying — zero ETL ingestion. Leveraging DuckDB's file ecosystem, you can query local or remote files (Parquet, CSV, JSON, S3, and more) directly in SQL, then map the results into DolphinDB distributed tables automatically. The data pipeline shortens dramatically.
Who benefits immediately
- Quantitative researchers — load Parquet market data files directly, explore locally, push results into backtesting with a single command
- Data engineers — unify heterogeneous data sources into a DolphinDB warehouse without format wrangling
- Analysts — export large computation results to DuckDB for deep local analysis
- Architects — build DolphinDB + DuckDB hybrid systems with clean hot/cold data separation
Where It Almost Broke
The toughest challenge wasn't the API integration. It was data type fidelity — specifically, preserving nanosecond timestamp precision across engine boundaries.
In financial data, a truncation error at the nanosecond level isn't a rounding issue. It's wrong data. Boye spent significant time working through the edge cases, iterating on the type mapping layer until every data type — including high-precision timestamps — aligned correctly across both systems.
"When that first piece of data crossed engines without any loss, I knew the core problem was solved."
On the development experience side, Boye was candid about what made the project feasible as an independent developer: DolphinDB's plugin development documentation is thorough, the open interfaces are well-designed, and plugins load as dynamic libraries — meaning the core system never needs to be modified, and updates to the plugin are fully isolated from the main installation. When he hit technical walls, the official team and community responded quickly.
The Bigger Picture
Boye's plugin is live and actively maintained. His roadmap: continued performance improvements, broader file format support, expanded SQL compatibility, and more plugins that connect DolphinDB to the wider data tooling ecosystem.
His message to other developers sitting on a useful idea:
"Start from real pain. Whether you're an independent developer or on an engineering team — if you've hit a friction point and have an idea for fixing it, build it. The barrier to entry is low, the support is there, and every small plugin adds genuine value to the ecosystem. Your niche frustration is probably someone else's daily blocker."
Try the DuckDB Plugin on the DolphinDB Plugin Marketplace.
Interested in building for the DolphinDB ecosystem? We'd love to hear from you.