Managing High-Resolution Weather Grid Data at Scale
Numerical weather prediction is increasingly moving from kilometer-scale grids toward resolutions as fine as 100 meters. Finer grids, longer forecast horizons, and more meteorological variables capture smaller-scale atmospheric changes, and the data is now used well beyond forecasting: in energy, agriculture, transportation, and disaster preparedness.
That precision comes at a price. A single high-resolution forecast can contain tens or even hundreds of millions of grid points. Multiply that by initialization time, forecast lead time, latitude, longitude, variable, and vertical level, then add years of accumulated history, and volumes reach terabytes and quickly approach petabytes.

The Challenge Goes Beyond Storage Capacity
1. Storage costs grow with resolution and history
Weather data is typically distributed in GRIB2, NetCDF, or HDF. They are primarily designed for data representation and exchange, while repeated analytical queries may benefit from database-oriented storage. To make the data queryable, teams usually load it into a database or another storage system, which can add storage overhead of its own.
As resolution increases and archives grow, costs rise with them. At terabyte to petabyte scale, reducing storage cost without sacrificing data precision becomes a core infrastructure requirement.
2. One data model rarely serves every query pattern
Users query weather grids in very different ways. They may retrieve a complete field for a given forecast, extract a geographic subregion, or look up a single grid point. These queries are often combined with conditions on initialization time, lead time, and variable.
Traditional databases handle structured queries well, but weather grids have an inherent two-dimensional spatial structure. Flattening them into row-per-point records inflates storage and makes it hard to get consistently good performance for both single-point lookups and full-field retrievals.
3. Moving data between systems adds cost and latency
Weather data exists to support forecasting and analysis, not just to be archived. Aggregating across forecast horizons, analyzing historical trends, detecting anomalies, and joining weather with power, agricultural, or logistics data all involve processing large volumes of data.
In many architectures, storage and computation live in separate systems: data sits in files, databases, or object storage, and a different engine performs the analysis. As volumes grow, reading, converting, and transferring data before every computation becomes expensive and puts pressure on network bandwidth and memory.
Adding yet another storage system or compute engine does not solve this. A better approach is to organize data according to its natural structure, let queries locate the data they need directly, and run computation close to where the data lives.
DolphinDB: Storage and Computation in One Platform
DolphinDB combines flexible data organization, partitioning, compression, and in-database computing, which reduces repeated data transfers between systems and addresses the specific demands of meteorological workloads.
TSDB engine with lossless compression
DolphinDB’s time-series storage engine (TSDB) combines partitioning, sorting, and compression for efficient storage and retrieval of large time-series datasets. Because precision matters in meteorology, the solution uses lossless compression: storage overhead is reduced while every original value is preserved.
Array vectors: grids stored in their natural shape
A weather grid is naturally two-dimensional. DolphinDB’s Array Vector data type allows multiple values to be stored in a single column entry. When weather grid data is organized by latitude, a row can hold the longitude values associated with that latitude.
This layout supports both access patterns. A full field is retrieved by reading the relevant rows. A single grid point is retrieved by locating the row for its latitude and then slicing the array at the corresponding longitude index. This can reduce the need for separate data representations for different query patterns.

Partition pruning: reading only what a query needs
Weather archives are multidimensional and keep growing. DolphinDB lets you partition data by dimensions such as initialization time, forecast lead time, and variable, depending on your workload. At query time, the engine uses the filter conditions to skip irrelevant partitions, so queries that align with the partitioning scheme scan only a fraction of the data, even as the archive grows.
DolphinDB’s distributed architecture also lets you scale storage and compute by adding nodes, leaving headroom for higher resolutions and larger volumes.
In-database computing: less data movement
DolphinDB provides more than 2,000 built-in functions and vectorized execution, so filtering, aggregation, and statistical analysis run directly inside the database. For example, temperature across multiple forecast horizons can be aggregated without exporting raw records to a client application. Because real-time and historical data live on the same platform, you can also run trend analysis, anomaly detection, and cross-source analytics without moving data between systems.
The following query computes a six-observation moving average of temperature for each grid point, ordered by forecast lead time. (context by groups rows by latitude and longitude, and csort sorts within each group before the window function is applied.)
select
latitude,
longitude,
forecastHour,
temperature,
mavg(temperature, 6) as temp_avg
from weather_grid
where initTime = 2025.03.20T00:00:00
context by latitude, longitude
csort forecastHourThe calculation runs entirely inside DolphinDB, with no need to transfer the dataset to an external compute engine.
Benchmark Results
We evaluated DolphinDB using a 1 km-resolution weather grid dataset containing 72 forecast lead times. The original dataset was approximately 500 GB, and the tests were conducted on a cluster of three standard servers.
Ingestion Performance

Query latency
Average response times under 50 concurrent clients:

These results demonstrate DolphinDB's ability to ingest and retrieve high-resolution weather grid data while supporting multidimensional queries and concurrent access.
Results reflect the specific dataset, hardware configuration, and test conditions described above. Actual performance may vary depending on data layout, query patterns, and system configuration.
From Storage to Decisions
Weather data is only valuable if it can be turned into decisions. With ingestion, storage, and analytics on one platform, large meteorological datasets can be stored efficiently, queried quickly, and processed where they sit. The same platform can then join weather data with operational data from energy, agriculture, and transportation systems.
Energy and utilities offer several clear examples:
- Renewable generation forecasting: Combine wind speed, solar irradiance, temperature, and other variables to forecast wind and solar output.
- Load forecasting: Use temperature, humidity, and other conditions to improve regional and sub-regional demand forecasts.
- Extreme weather and grid risk: Assess operational risk from heat waves, heavy rainfall, tropical cyclones, wildfire conditions, and icing events.
- Trading and dispatch: Feed generation and load forecasts into day-ahead and intraday trading and dispatch decisions.
By consolidating ingestion, storage, computation, and analytics on one platform, DolphinDB cuts data movement and integration work across systems that are usually fragmented. High-resolution weather data can then feed directly into business analytics workflows.