Gemini 3 Just Set a New Benchmark: 174% Faster Factor Replication
Factor replication from research reports has traditionally been a core method for analysts seeking to expand their analytical frameworks and identify new alpha-generating opportunities. Conventional approaches involve painstaking manual processes—reading reports line by line and hand-coding investment logic—which are time-intensive, resource-heavy, and susceptible to human error and oversight. As large language models have advanced, the quantitative finance industry has begun exploring automated report-based factor extraction. Yet challenges including inaccurate code generation and lengthy processing times have confined these solutions to pilot testing rather than robust production deployment.
Developed by DolphinDB, the Starfish Research Reports AI Assistant is an intelligent platform designed to streamline research-driven factor mining and analysis. Leveraging the high-performance DeepSeek model, it seamlessly integrates natural language understanding with quantitative research workflows, delivering a comprehensive end-to-end solution: upload research report → extract investment factors → generate executable code → conduct backtesting → deliver actionable insights. This tool provides quantitative researchers with unprecedented efficiency and precision in AI-assisted analysis.
Despite strong initial performance, deploying Starfish AI in production environments revealed critical bottlenecks:
- Extended iteration cycles — earlier iterations required an average of 6.8 rounds to generate executable code
- Limited code accuracy — only approximately 20% of generated code correctly captured the underlying research logic and executed successfully
- Semantic misinterpretation — natural language ambiguities in reports led to incomplete factor identification
Following the November release of Gemini 3 , integrating this advanced model into the Starfish AI platform yielded remarkable improvements. We evaluated performance across 20 diverse research reports, assessing critical dimensions including factor reproduction fidelity, code generation quality, and logical accuracy.
The results demonstrate substantial advancement:
- Syntax correctness: 97% of generated factor code compiles and runs without errors
- Logical accuracy: 50% of code precisely matches the intended research methodology
- Iteration efficiency: 174% improvement in the average number of cycles required
These metrics represent a significant leap beyond previous model iterations. In the following sections, we'll explore the technical details and implementation insights behind these achievements.
Real-World Evaluation: Gemini 3’s Impact on Starfish AI
The Starfish Research Reports AI Assistant enables users to upload PDF research reports directly, with the system autonomously handling factor extraction and code generation from end to end.
Evaluation Methodology
To establish rigorous performance benchmarks, we selected DeepSeek V3.1—a model recognized for robust overall capabilities—as our comparative baseline. After optimizing prompts and workflows specifically for DeepSeek V3.1, we curated a test set of 20 out-of-sample research reports spanning diverse investment strategies including momentum, value, and event-driven approaches. Under controlled conditions—identical hardware infrastructure, data sources, and backtesting frameworks—we conducted parallel evaluations of Gemini 3 and DeepSeek V3.1, measuring critical performance indicators such as factor extraction completeness, iteration requirements, and code accuracy.
Test Results: Gemini 3 Delivers Comprehensive Performance Gains
The evaluation reveals that Gemini 3 achieves substantial improvements over DeepSeek V3.1 across both fundamental dimensions of factor mining: success rate and operational efficiency.
Factor Coverage Completeness
Gemini 3 identified 247 factors across the 20 research reports, successfully extracting 96% of all discoverable factors (247 out of 250). By comparison, DeepSeek V3.1 captured only 76% of the factor universe—a 20-percentage-point gap indicating significantly superior comprehension and extraction capabilities.
Code Generation Performance
Of the 247 extracted factors, Gemini 3 generated executable, error-free code for 240 factors—a 97% success rate. This represents an 8-percentage-point improvement over DeepSeek V3.1's 89% rate (172 out of 192 factors), effectively eliminating the risk of task-level failures in production workflows.
Iteration Efficiency
Gemini 3 dramatically reduced the iterative refinement burden, requiring an average of just 2.33 iterations per factor compared to DeepSeek V3.1's 6.3 iterations—a 174% efficiency improvement that substantially accelerates research timelines.
True Factor Reproduction Fidelity
Generating syntactically correct code represents only the initial threshold; accurately reproducing the underlying research logic demands rigorous validation across multiple dimensions including data alignment, parameter calibration, and methodological fidelity. In a manual verification study sampling one factor from each of 15 reports:
- Gemini 3 achieved approximately 50% logical accuracy —meaning the generated code correctly implemented the intended research methodology
- DeepSeek V3.1 achieved approximately 20% logical accuracy
This 2.5× improvement in true reproduction accuracy represents a fundamental advancement in the practical utility of AI-assisted factor research.
Comparison of Gemini 3 vs. DeepSeek V3.1 on Research-Report-Based Factor Mining
| Research Report Title | Gemini 3 – Extracted Factors | Gemini 3 – Successful Code | Gemini 3 – Avg. Iterations | DeepSeek V3.1 – Extracted Factors | DeepSeek V3.1 – Successful Code | DeepSeek V3.1 – Avg. Iterations |
|---|---|---|---|---|---|---|
| Commodity Trend Series: Rebuilding Commodity Momentum Strategies | 8 | 8 | 1.625 | 4 | 4 | 6.25 |
| Financial Engineering Research: A Unified Framework of Momentum & Reversal from a Risk-Premium Perspective | 11 | 11 | 2.18 | 9 | 6 | 7.69 |
| UMR 2.0: Upgraded Unified Framework of Momentum–Reversal from a Risk-Premium Perspective | 49 | 48 | 2.06 | 9 | 8 | 6.11 |
| Quantitative Investment: Mining Alphas from Massive Technical Indicators | 66 | 66 | 1.33 | 63 | 63 | 5.24 |
| Alpha Information Hidden in Volume Surges | 7 | 7 | 1.28 | 7 | 4 | 9 |
| Tidal Patterns in Single-Stock Trading Volume and Construction of the “Tide” Factor | 6 | 6 | 3.6 | 6 | 6 | 6 |
| Changes in Single-Stock Volatility and the “Scaling New Heights” Factor | 10 | 10 | 1.6 | 9 | 4 | 15 |
| Identifying Single-Stock Momentum Effects and the "Momentum-Reversal Regime" Factor | 11 | 11 | 1.45 | 12 | 10 | 5 |
| Volatility of Volatility and Investor Ambiguity Aversion | 6 | 6 | 1.33 | 6 | 6 | 6 |
| Single-Stock Price Jumps and Improvements to the Amplitude Factor | 9 | 9 | 2.5 | 9 | 6 | 11.33 |
| Significant Effects, Extreme Return Distortions, Decision Weights, and the "Overreaction" Factor | 17 | 17 | 2 | 9 | 9 | 9.11 |
| Market Follow-Through of Single-Stock Turnover and the "Flow-Following" Factor | 4 | 4 | 2.25 | 4 | 4 | 1.5 |
| Behavioral Finance Perspective: The “Salience Return” Factor (STR) | 4 | 4 | 3.75 | 4 | 4 | 3.25 |
| Applications of the Salience Theory to Price & Volume in A-Shares | 4 | 4 | 4.5 | 4 | 4 | 4.75 |
| Effective Industry Volume-Price Factors and Industry Rotation Strategies | 11 | 11 | 1.9 | 11 | 11 | 3.8 |
| Constructing Momentum Factors in the A-Share Market | 6 | 5 | 7.66 | 8 | 8 | 4.25 |
| Fund Overweighting Factors and Their Impact on Index-Enhancement Portfolios | 3 | 3 | 2.33 | 3 | 3 | 3.66 |
| Advanced Version of the APM Factor Model | 7 | 4 | 8.85 | 7 | 7 | 8.28 |
| Pure Volatility Factor After Removing Cross-Period Cross-Section Correlation | 5 | 3 | 7.4 | 5 | 4 | 5.8 |
| Measuring Buying/Selling Pressure from Price–Volume Relations | 3 | 3 | 3 | 3 | 3 | 2 |
| Total/Average | 247 | 240（97.17%） | 2.33 | 192 | 172（89.58%） | 6.3 |
Detailed inspection reveals that Gemini 3's primary advantage lies in its rigorous adherence to source material. During the factor-formula reproduction stage, it faithfully replicates the mathematical expressions presented in research reports, avoiding DeepSeek's frequent tendency to "oversimplify" or reinterpret formulas. When generating executable code, Gemini 3 achieves higher success rates while maintaining close logical alignment with the original reports. The most common remaining errors involve unfamiliarity with specific DolphinDB function signatures or parameter conventions, which are typically resolved within 1-2 iterations.
Comparative Case Studies
| # | Factor | Gemini 3 | DeepSeek V3.1 |
|---|---|---|---|
| 1 | Industry Effective Volume–Price Factor — first_order_divergence | Formula: Accurately captures the "price-volume relationship + rank + mcorr" methodology described in the report. Code: Generated correctly on first attempt. Properly implements the mcorr function with concise, well-structured expressions. | Formula: Produced a logically inconsistent formula that deviates from the report. Code: Failed initial generation. After 5 iterations, produced runnable code that still mismatched the intended formula—grouping by time bar, rank operations, and mcorr logic were all missing. |
| 2 | “Scaling New Heights” Factor | Formula: Correctly implements the multi-step logic: compute minute-by-minute volatility from close prices → select high-volatility minutes → compute covariance → apply 20-day rolling mean minus standard deviation. Code: Generated successfully. After 3 iterations, final version closely matched report specifications and met production-readiness standards. | Formula: Incorrectly computed volatility from a single OHLC bar rather than the close-price time series. Code: Failed after 15 iterations. Misinterpreted "volatility" as single-bar four-price volatility; regression and factor-composition logic substantially deviated from the report. |
Despite strong overall performance, Gemini 3 exhibits occasional weaknesses:
- Suboptimal function usage: In complex scenarios, may introduce unnecessary rolling window operations or fail to leverage DolphinDB's native functions like mbeta and mcorr optimally, instead defaulting to more verbose computational approaches.
- Grouping logic misinterpretation: Occasionally misunderstands nuanced grouping constructs such as context by + interval operations.
For these edge cases, a hybrid approach remains essential: human experts provide the final creative judgment, perform data-alignment verification, and conduct risk validation, while the AI model handles the computationally intensive tasks of report parsing, formula extraction, and initial code scaffolding. This division of labor maximizes both efficiency and accuracy in production environments.
Dlang Code Generation: Model Improvements Driving Superior Tool Performance
The core capability of the Starfish Research Reports AI Assistant centers on transforming natural-language factor descriptions into high-performance, executable Dlang code. DolphinDB's proprietary scripting language, Dlang, is renowned for its computational speed and vectorized operations. Consequently, the underlying large language model's comprehension and generation capabilities directly determine the quality of the end-user experience.
Following the transition to Gemini 3, we conducted rigorous benchmarking on Dlang code generation accuracy. Across 1,481 test problems spanning diverse financial computation scenarios, Gemini 3's code-logic correctness improved from 17% (DeepSeek R1) to 34%—a breakthrough that fundamentally underpins Starfish AI's performance gains. In practical terms, Gemini 3 now produces directly executable, logically correct code approximately one-third of the time, while in remaining cases it still delivers highly usable logical frameworks that require minimal refinement.
Critically, 34% does not represent a theoretical ceiling. As DolphinDB continues to incorporate Dlang best practices and domain-specific financial computation patterns into the model's training corpus and knowledge base, this metric is expected to rise further. The tool's productivity leap ultimately derives from an enhancement in the underlying model's "cognitive horsepower."
Starfish AI: A New Paradigm in Intelligent Research
The integration of Gemini 3 has substantially strengthened the Starfish Research Reports AI Assistant's factor-mining capabilities through deeper semantic understanding, more accurate code logic, and dramatically improved iteration efficiency. However, this advancement represents just one dimension of Starfish AI's comprehensive capability framework.
Starfish AI is DolphinDB's end-to-end solution for quantitative research and investment workflows. It encompasses the complete lifecycle: factor computation, evaluation and analysis, strategy backtesting, performance attribution, and workflow orchestration—creating a seamless closed loop from factor discovery through strategy execution. Building upon this foundation, its AI-powered feature set includes automated factor code generation, one-click conversion of strategy logic, and intelligent authoring of data analysis scripts.
DolphinDB's deep integration with industry-leading large language models has fundamentally expanded the efficiency frontier for quantitative research. Starfish AI is currently available for trial access to professional financial institutions.
Try it now: https://dolphindb.com/product#starfish