For quantitative research, the first challenge is often the data, not the strategy.
Where do you get market data? How do you load years of history in bulk? How do you store different asset classes consistently? And once new data arrives, how do you keep your database up to date?
Tushare is a free, open-source Python package for accessing financial data, covering stocks, funds, futures, bonds, and indices in the Chinese market. But getting the data is only the first step. API calls, table creation, bulk ingestion, and daily incremental updates all have to be built before you have a working data pipeline. If you develop and maintain each of these yourself, ingestion becomes an ongoing engineering burden as your research expands.
To remove that burden, DolphinDB offers EasyTushare, a module that connects Tushare to DolphinDB and automates the whole workflow: database and table initialization, bulk historical ingestion, and scheduled daily updates. Set it up once, and your data stays current and ready for querying, computation, and research.
Requirements: EasyTushare 1.0.0 requires DolphinDB 3.00.6 or later and the httpClient plugin, which you need to install and configure first.
One Module for Multiple Data Categories
EasyTushare organizes ingestion by asset class. It currently covers 10 data categories: stocks, ETFs, mutual funds, futures, bonds, indices, options, foreign exchange, spot gold, and corpus data for large language models (LLMs). Macroeconomic data is also supported.
For stocks, the available data goes well beyond basic information and daily prices. It includes minute-level K-lines, adjusted prices, financial statements, capital flows, top-list (Dragon and Tiger List) data, margin trading and short selling data, and more. Futures, bonds, indices, and options similarly offer basic information, market data, and asset-specific datasets.
Instead of building separate ingestion logic for each asset class, you select the tables you need and load them through the same workflow.
From Historical Loading to Automated Daily Updates
Setting up a financial database involves three stages: initializing databases and tables, loading historical data, and keeping the data up to date.
1. Initialize the databases and tables
EasyTushare does more than wrap Tushare API calls. It provides a consistent ingestion framework through three core functions:
- Init: Initializes an asset category, including table creation and bulk data loading.
- CreateTable: Defines table schemas and provides table creation functions.
- AppendData: Retrieves data through the Tushare API and writes it to DolphinDB distributed tables.
By default, EasyTushare stores data in three DolphinDB distributed databases: one for basic information and fundamentals, one for daily and periodic market data, and one for minute-level K-lines. Daily data is partitioned by year, and minute-level data by day.
To initialize the tables, call the corresponding functions:
// Create databases
createDb()
// Create stock data tables
stockInitTables()2. Load historical data in bulk
Once the databases and tables are ready, you can load historical data. EasyTushare provides loading functions for each asset class. You specify a date range and, where supported, the tables to import. Ingestion runs as background tasks.
For example, this loads stock data for 2024:
use EasyTushare::stockInit
// Load historical stock data for 2024
stockHistoryDataLoad(
token,
2024.01.01,
2024.12.31
)To skip data you do not need, specify the tables. This call loads only basic stock information and daily market data:
// Load only basic stock information and daily market data
stockHistoryDataLoad(
token,
2024.01.01,
2024.12.31,
["stock_basic", "stock_daily"]
)The same approach works for selected intraday data, such as one-minute or five-minute K-lines, where the relevant ingestion functions support it.
3. Keep the database up to date
After the historical load, the database needs to keep up with new market data. EasyTushare provides scheduled daily ingestion functions for each asset class. Register a scheduled job, and the latest daily data is retrieved and written to DolphinDB automatically.
use EasyTushare::stockSchedulejob
// Schedule daily ingestion of stock data
scheduleJobStockDataDaily(token)Conclusion
EasyTushare simplifies ingestion between Tushare and DolphinDB. If you are setting up a new research environment, you can use it to initialize and populate your databases. If you already use DolphinDB, you can add only the datasets your research needs.
Set it up once, keep your data current, and spend your time on analysis and strategy research instead of maintaining pipelines.