V2.00.18 & 3.00.5 Are Live! Major TPC-H Gains, the Arrival of the MPP Framework…

DolphinDB 2.00.18 and 3.00.5 are now officially released!

This release is centered on three pillars: speed, stability, and usability. Query and analytics performance has been substantially elevated, distributed computing has received a comprehensive overhaul, and resource scheduling under the storage-compute separation architecture is now more intelligent and fine-grained than ever.

Beyond core infrastructure, this update brings meaningful enhancements to streaming analytics, financial pricing, industrial monitoring, and other mission-critical use cases.

Read on as we walk through seven key improvements that define this release.

Key Enhancements

Major Upgrade to the SQL Execution Engine

The following features are supported in version 3.00.5 only.

This release delivers two landmark advancements in SQL execution:

• Substantially improved performance for single-node Join operations
• The introduction of a native MPP (Massively Parallel Processing) framework for distributed parallel computing

Significant Join Performance Improvements

Version 3.00.5 brings deep, targeted optimizations to the SQL execution engine, resulting in broad single-node performance gains. Central to this upgrade is an extensive refactoring of multi-table Join execution logic and the Hash Join operator—two of the engine's most critical components. The result is smarter execution plan selection and markedly higher Join efficiency.

In TPC-H benchmark testing, the new version demonstrates notable performance gains over its predecessor in single-node environments. Analytical queries involving complex joins and aggregations—such as multi-table Join workloads typical of TPC-H—see the most pronounced improvements.

Introduction of the MPP (Massively Parallel Processing) Framework

This release also debuts a brand-new MPP framework that fundamentally reimagines how complex queries are executed. Rather than relying on centralized computation, the framework intelligently decomposes demanding workloads—particularly multi-table analytical queries—and dynamically distributes them across multiple cluster nodes for parallel execution. This unlocks the full potential of distributed computing resources.

The performance impact is substantial. In TPC-H benchmark testing, enabling the MPP engine produces breakthrough results for typical multi-table join and analytical workloads. In complex business analytics scenarios such as TPC-H Q5 and Q8—which involve multi-table joins and large-scale aggregations—MPP delivers performance gains ranging from several times to more than an order of magnitude. In some cases, query execution time drops from minutes to seconds.

This architectural leap positions DolphinDB to efficiently handle large-scale ad-hoc analytics, real-time risk control, and complex report generation at enterprise scale, significantly strengthening its foundation for business intelligence and data-intensive analytical workloads.

Precise Caching and Intelligent Compression in Storage-Compute Separation

The following features are supported in version 3.00.5 only.

This release brings significant enhancements to the storage-compute separation architecture, with a focus on two areas: refined cache management and efficient data transfer. Together, these improvements make resource scheduling more precise and flexible, elevating distributed computing performance and intelligent resource utilization to a new level.

Compute-Group-Level Table Cache Rule Configuration

Three new functions—addCacheRulesForComputeGroup, removeCacheRulesForComputeGroup, and getCacheRulesForComputeGroup—give administrators granular control over which distributed tables are cached for specific compute groups.

This marks a shift from a blunt "cache everything" approach to a targeted, on-demand caching model. Memory resources can now be strategically directed toward high-frequency, business-critical tables, meaningfully reducing I/O latency and boosting query performance for priority workloads.

Cache rules are also integrated with table lifecycle management and governed by access control policies, enhancing both controllability and security. The result is a caching strategy that strikes the right balance between cost and performance.

Improved Data Transfer Efficiency in Storage-Compute Separation

This release introduces compression control for data transfers between data nodes and compute nodes. The feature can be persistently enabled or disabled via the enableTransferCompressionToComputeNode configuration parameter, or toggled dynamically at runtime using the function of the same name.

This flexibility allows administrators to tune transfer compression in response to real-time network conditions and workload characteristics, striking an optimal balance between network bandwidth consumption and CPU overhead.

The practical benefits are significant: reduced cross-node network pressure, shorter data readiness times, and improved overall throughput across the query and computation pipeline. This feature is especially valuable in geographically distributed clusters or bandwidth-constrained environments.

Comprehensive Evolution of the ORCA Platform

The following features are supported in version 3.00.5 only.

This release delivers a systematic round of functional enhancements and architectural upgrades to the ORCA real-time computing platform.

Improved Usability: Simplified UDF Engine Creation

The UDF Engine creation process has been significantly streamlined. Users can now focus entirely on writing their function logic, while the system handles the rest—output schemas are inferred automatically, external variables no longer require manual initialization, and one-step integration enables rapid deployment. This substantially lowers the barrier to developing and deploying real-time custom computation logic.

Strengthened Security: Unified Permission Control

This release introduces a comprehensive access control framework for ORCA. Core streaming resources—including stream graphs and stream tables—are now fully integrated into the unified permission management system. Functions executed within a UDF Engine strictly inherit the creator's execution privileges, and any unauthorized operations are actively blocked at both creation time and runtime. This provides strong compliance guarantees for collaborative development and production environment governance.

Enhanced Reliability: Introduction of Shared Objects

The new version introduces shared objects as a foundational reliability mechanism, including shared in-memory tables (DStream::sharedTable), shared keyed tables (DStream::sharedKeyedTable), and shared dictionaries (DStream::sharedDict).

Where external variables were previously lost on restart or invisible across nodes, shared objects are now centrally managed by ORCA. This enables persistent state recovery, cross-Engine and cross-Task collaborative access within the same stream graph, and high-performance incremental checkpointing for significantly faster state restoration after failures—laying a solid foundation for high-availability and distributed stream processing scenarios.

Further Strengthening High Availability

The following features are supported in version 2.00.18/3.00.5.

This release brings substantial improvements to high availability across the platform on two fronts.

First, the publish-subscribe mechanism for high-availability stream tables has been fully refactored, significantly improving the stability and resilience of streaming data during failover scenarios.

Second, the new version introduces support for high-availability MVCC tables (created via the HaMvccTable function). Building on the familiar functionality and interface of MvccTable, the HA variant integrates the Raft consensus protocol at the underlying layer to deliver strong cross-node consistency, automatic failover, and reliable data replication—without any changes to the day-to-day user experience.

Getting started is straightforward: users simply specify the high-availability cluster configuration at table creation time to gain transparent data replication and automatic recovery. All standard data operations—insert, update, delete, and query—remain fully consistent with existing interfaces, keeping the learning curve minimal.

Streaming Data Updates and State Recalculation

The following features are supported in version 3.00.5 only.

This release introduces a landmark capability to the streaming engine: native support for Update operations. This represents a meaningful evolution in stream processing, enabling pipelines to respond precisely to dynamic data changes in a way that closely mirrors how batch systems handle data modifications.

Native Support for Update Semantics

Streaming engines have traditionally been built around append-only data models. When business logic requires updates to existing keys—such as in-progress candlestick (K-line) data or the latest sensor readings—teams have historically relied on cumbersome workarounds. This release eliminates that complexity by enabling the engine to natively understand and process the semantic intent of data updates. This capability is currently supported by selected core operators within the Reactive State Engine and the Time-Series Aggregation Engine.

State-Aware Stream Tables

To underpin update semantics, this release introduces a new state stream table type created via the changelogStreamTable function. The table uses built-in operation flags—N (New/Insert) and U (Update)—to communicate data operation intent to the engine through a unified interface.

When the engine encounters a U (Update) record, it executes a precise two-step process: first rolling back the influence of the previous value for the given key, then recalculating based on the new value. This mechanism preserves correctness and continuity in stateful computations such as cumulative sums and time-window aggregations, ensuring that real-time metrics are automatically and accurately revised whenever underlying data changes.

Additionally, the peekAppend function offers a sandbox-style testing environment, allowing users to validate the effects of data changes before committing updates to production logic.

Designed for Real-World Dynamic Scenarios

This feature is purpose-built for use cases with stringent requirements around timeliness and accuracy, including real-time K-line indicator calculations, sensor state monitoring, and real-time risk metric updates. By enabling stream pipelines to handle continuously evolving data naturally and reliably, this upgrade significantly strengthens the robustness and real-world applicability of DolphinDB's streaming solutions.

New FICC Capabilities

The following features are supported in version 3.00.5 only.

This release introduces two core streaming engines tailored for FICC scenarios: the Real-Time Curve & Surface Construction Engine (Market Data Engine) and the Real-Time Pricing Engine. Together, they form a highly automated, end-to-end solution for real-time market data processing and financial instrument pricing.

Real-Time Curve & Surface Construction Engine (Market Data Engine)

The Market Data Engine is designed to automatically transform fragmented raw quotes and market data into fully constructed market curves—such as yield curves—and surfaces—such as volatility surfaces—in real time.

Traditionally, preparing market data for valuation has been a manual, time-consuming process that frequently lags behind live market conditions. This engine eliminates that bottleneck entirely. Trading and risk management teams gain immediate access to up-to-date market curves, data preparation efficiency is dramatically improved, and valuation inputs consistently reflect the latest market state.

Real-Time Pricing Engine

The Pricing Engine automatically triggers the repricing of financial instruments—such as bonds and derivatives—whenever underlying market data changes, such as shifts in interest rate curves.

This directly addresses the long-standing challenge of valuation latency. With millisecond-level responsiveness, the engine supports end-of-day batch valuation, intraday real-time mark-to-market, and rapid decision-making under volatile market conditions—keeping pricing calculations continuously synchronized with live markets.

End-to-End Real-Time Processing Loop

The two engines are designed to work in concert. Real-time constructed market curves feed directly into the Pricing Engine, forming a seamless closed-loop pipeline: Raw Quotes → Structured Market Data → Instrument Pricing. The result is a complete, fully automated real-time valuation workflow.

Enhanced Event Monitoring Engine for Industrial Scenarios

The following features are supported in version 2.00.18/3.00.5.

This release introduces two targeted enhancements to the streaming engine for industrial IoT workloads. Both upgrades are designed around the unique characteristics of industrial data, enabling more precise and lower-overhead processing of complex device event streams.

Sparse Reactive State Engine

The newly introduced Sparse Reactive State Engine is purpose-built for processing stateful events that depend on historical windows—for example, detecting when a condition has been satisfied three consecutive times.

Its defining innovation is sparse computation. In industrial environments, a single device or production line may be associated with hundreds of sensors, yet at any given moment only a small fraction of them generates meaningful data. Traditional state engines tend to perform unnecessary computations across large numbers of null or inactive fields, leading to wasted processing overhead.

The Sparse Reactive State Engine resolves this by allowing independent rule configuration per sensor metric. Computation is triggered only when relevant, valid data arrives, and only the rules associated with that data are executed. This fundamentally eliminates the performance penalty of null-value processing and aligns naturally with the sparse characteristics of industrial data streams. The engine can be instantiated via createSparseReactiveStateEngine or DStream::SparseReactiveStateEngine under the ORCA framework.

Enhanced Reactive Stateless Engine

The Reactive Stateless Engine handles stateless events based on current-state logical combinations—for instance, evaluating "when condition A and condition B are both true, check the value of C."

The key enhancement in this release is the introduction of the triggerOn parameter, which gives users precise control over which metrics trigger the evaluation of each composite event. A final alert condition, for example, can be configured to fire only when specific key indicators are updated, preventing unrelated state changes from causing unnecessary or false triggers. This enables fine-grained control over computation behavior while further reducing overhead.

Full Lifecycle Rule Management

Complementing both engines, newly added rule management functions allow users to dynamically add, remove, and query monitoring rules throughout their entire lifecycle—without interrupting running pipelines.

Taken together, these enhancements equip the system to process complex event streams in industrial production monitoring and predictive maintenance scenarios with lower resource consumption, cleaner logical definitions, and higher execution precision. They provide enterprises with a powerful, maintainable foundation for building high-availability real-time monitoring systems.

Other Enhancements

A Richer and More Powerful Function Library

New FICC & Equity Functions

Supported in version 3.00.5 only.

Alongside the Real-Time Curve/Surface Construction Engine and Real-Time Pricing Engine, this release introduces a comprehensive suite of standardized utility functions spanning foundational financial calculations to professional pricing models.

Core Utility Functions

On the foundational side, the newyearFrac function now supports multiple day count conventions—including Actual/360, Actual/365, and Actual/Actual ISDA/ISMA—for accurate year fraction computation between two dates under specified accrual rules. The instrumentPricer and portfolioPricer functions have been extended to accept Market Data Engine and UDF as input sources, while parseInstrument now supports a broader range of financial instrument types. Rounding out the core additions, plot gains support for 3D surface charts, enabling intuitive visualization of three-dimensional financial data such as volatility surfaces.

Fixed Income Enhancements

bondInstrumentCalculator has been upgraded to support exchange-traded option-embedded bonds (Option Bonds), with algorithms aligned to CSI (China Securities Index) standards.

Financial Engineering Functions

For equity markets, this release adds a full set of functions covering the end-to-end workflow of equity derivatives: eqDividendCurveBuilder, eqVolatilitySurfaceBuilder, eqEuropeanOptionPricer, eqAmericanOptionPricer, and eqProxyVolatilitySurfaceBuilder—supporting dividend curve construction, volatility surface building, and option pricing.

For commodity markets, three new functions—cmFutVolatilitySurfaceBuilder, cmFutAmericanOptionPricer, and cmFutEuropeanOptionPricer—extend volatility surface construction and option pricing capabilities to commodity futures.

The following pricing functions now support batch pricing mode: bondPricer, bondFuturesPricer, irDepositPricer, irFixedFloatingSwapPricer, fxForwardPricer, fxSwapPricer, and fxEuropeanOptionPricer. All option pricing functions also support Greeks (risk sensitivities) calculation.

Time-Series Processing

Supported in versions 2.00.18 and 3.00.5.

movingWindowData now supports array vector, enhancing sliding-window analysis for complex structured data. The higher-order function contextBy has been extended to support custom function-based time-window sliding calculations, enabling more flexible rolling aggregation within groups—for example:

contextby(func=tmoving{udf,,,3d}, funcArgs=(trade_date,(price, qty)), groupingCol=sym)

Statistical Analysis & Data Operations

Supported in versions 2.00.18 and 3.00.5.

A new covarp function has been added alongside its moving-window and row-based variants, with full support within the streaming engine. For data binning, qcut enables quantile-based grouping suited to factor analysis and performance attribution. The new remove! function supports element removal from vectors or tuples by position.

Similarity & Distance Functions

Supported in versions 2.00.18 and 3.00.5.

Four new vector similarity and distance functions strengthen DolphinDB's capabilities in quantitative research and data science: minkowski (Minkowski distance), seuclidean (standardized Euclidean distance), cosine (cosine similarity), and mahalanobis (Mahalanobis distance).

Expanded Federated Query Capabilities

Supported in version 3.00.5 only.

This release further strengthens DolphinDB's database connectivity and federated query capabilities, broadening multi-source data integration and cross-system analytics.

External table creation now supports four additional data sources, including Hive and GaussDB, significantly simplifying access to heterogeneous data environments. On the federated query front, the new runExternalQuery function enables unified SQL passthrough access to mainstream databases including MySQL, PostgreSQL, Oracle, Hive, and GaussDB. Users can perform real-time cross-source queries and joint analysis without any data migration, enabling seamless multi-database analytics within a single unified framework.

More Robust SQL Syntax

Supported in versions 2.00.18 and 3.00.5.

This release continues to advance SQL programmability and standards compliance across several areas.

For metaprogramming, cgroup by semantics are now supported in function-based SQL meta-programming scenarios, allowing users to dynamically generate SQL queries with customized grouping logic. This significantly improves flexibility and expressive power in dynamic SQL generation. The insert into statement now supports partial-column insertion in line with standard SQL conventions, letting users explicitly specify target columns during insertion for greater write flexibility. Finally, the new matchedRowCount function returns the number of rows matched in the most recent update, delete, or merge operation, providing precise visibility into the scope of data modification operations.

Further Enhancements in Stream Processing

This release extends streaming data processing in two meaningful ways, improving flexibility for complex financial real-time aggregation and stream observability.

Array Vector Grouping in the Time-Series Engine

Supported in versions 2.00.18 and 3.00.5.

The time-series aggregation engine now supports Array Vector types as grouping fields. The system automatically flattens array vectors into one-dimensional vectors prior to grouping calculations—a particularly useful capability for complex market data structures such as multi-level order book quotes and tick-by-tick trade data.

Within a single time window, users can perform real-time aggregation across multiple dimensions simultaneously—such as symbol, side (buy/sell), and order price level—enabling real-time computation of metrics like order amount and count per price level. This significantly simplifies high-frequency market data processing logic.

Streaming SQL Metadata Query

Supported in version 3.00.5 only.

The new getStreamingSQLSubscriptionInfo function allows users to query metadata for a specified streaming SQL task. It returns a dictionary containing three key fields: leadingRecordInsertTime (the earliest data insertion time that triggered the most recent update), lastRecordInsertTime (the latest data insertion time that triggered the most recent update), and lastUpdateTime (the most recent update time of the current result set). This enhancement provides finer-grained observability for monitoring, debugging, and operating streaming computation tasks.

Improved Operational Efficiency and Security

This release delivers a range of enhancements across observability, access control, and license management, further strengthening cluster operations efficiency and security.

Function View Management Enhancement

Supported in versions 2.00.18 and 3.00.5.

getFunctionViews now returns additional metadata fields—Owner (the creator of the function view) and Create Time—making it easier to trace the origin and ownership of custom function views. This strengthens code governance and auditability across teams.

Refined Access Control for Distributed Tables

Supported in Server versions 2.00.18 and 3.00.5.

getTables now enforces stricter permission controls. Non-administrator users can only retrieve distributed tables for which they hold explicit database- or table-level permissions, such as DB_READ or TABLE_READ. This prevents unintended exposure of sensitive metadata and improves data isolation in multi-tenant environments.

Simplified License Management

Supported in Server versions 2.00.18 and 3.00.5.

The commercial license server now supports online updates to client license expiration times via the setLicenseServerUserResource function—no service restart required. This reduces operational overhead and streamlines routine maintenance.

Optimized Cluster Object Queries

Supported in Server version 3.00.5 only.

getClusterDFSDatabases and getClusterDFSTables now include optional includeSysDb and includeSysTable parameters, allowing administrators to filter out system databases and tables during queries. This reduces noise in query results and improves the efficiency of daily cluster inspections.

Expanded Single Sign-On (SSO) Capabilities

Supported in Server version 3.00.5 only.

The SSO workflow now supports the HTTP protocol. A new configuration parameter, oauthAllowPasswordLoginNodes , enables administrators to control password-based login permissions at the node level, providing greater flexibility for deployments that require hybrid security strategies.

System-Level Optimizations

This release continues to raise the bar for usability, performance, and observability across the platform.

On the operations and monitoring front, getFunctionViews now surfaces creator and creation time metadata, getSessionMemoryStat adds support for primary-key engine cache statistics, and memory usage for persistent stream tables has been optimized.

For developer experience, syntax error messages now include line number positioning for faster debugging. Assignment statements and mutable functions have been extended to support direct modification of tuple and dictionary members, and API string writes now support automatic type conversion based on field types.

Performance improvements span several areas: the Window Join Engine and Nearest Join Engine have been enhanced, context by grouped query performance has been optimized, fuzzy matching in the text storage engine is faster, and cross-cluster query performance in ORCA has been further improved.

At the infrastructure level, foundational components—including the SSO workflow, License Server startup strategy, and write-volume allocation strategy—have been optimized to improve overall system stability and efficiency in complex production environments.

Upgrade Notes

Due to compatibility changes in permission files, users are strongly advised to back up all ACL-related files before proceeding with the upgrade. For Raft clusters, Raft-related files must also be backed up. The specific files required vary by deployment type, as outlined below.

• aclCheckPoint.meta
• aclEditlog.meta

Standard Cluster| < YourPath>/server/cluster/data/| < homeDir>/sysmgmt/

• aclCheckPoint.meta
• aclEditlog.meta

High-Availability Cluster| < YourPath>/server/clusterDemo/data/All controller nodes within the Raft group must be backed up.| < homeDir>/sysmgmt/

• aclCheckPoint.meta
• aclEditlog.meta

< homeDir>/raft/

• raftHardstate
• raftSnapshot
• raftWAL

For a complete list of optimizations, bug fixes, and compatibility updates included in this release, please refer to the full release notes in the Documentation Center.