Industrial Internet of Things (IIoT) and Internet of Vehicles (IoV) applications rely on sensor data to power predictive maintenance, equipment monitoring, automated control, and operational decision-making.

As connected devices continue to proliferate, the volume of sensor data has grown at an unprecedented rate. Industry reports estimate that connected industrial equipment in China generates more than 800 PB of time-series data every year. Even a mid-sized deployment with 25,000 sensors reporting once per second can produce over 430 GB of data per day, while large multi-site deployments easily accumulate hundreds of terabytes annually.

But data volume is only part of the challenge. Raw sensor readings are inherently noisy, and simply collecting more data does not necessarily lead to better decisions. The real challenge is transforming massive streams of imperfect sensor data into reliable, real-time insights.

Why Sensor Data Needs Filtering

Raw sensor readings are inevitably affected by environmental noise, electromagnetic interference, hardware limitations, and communication instability. These factors introduce random fluctuations, spikes, and outliers that often make raw measurements unsuitable for downstream analytics.

In industrial systems, filtering is the process of extracting the most reliable estimate of the true system state from noisy observations.

Typical objectives include:

Noise suppression – Remove random fluctuations and reduce false alarms.
State estimation – Combine historical measurements with mathematical models to estimate the current or even future operating state.
Sensor fusion – Integrate heterogeneous measurements (such as GPS, accelerometers, and gyroscopes) into a single, reliable estimate.

For modern IoT systems, filtering is not an optional optimization—it is a prerequisite for trustworthy analytics and real-time decision making.

The Cost of Traditional Filtering Pipelines

Although filtering is essential for Industrial IoT (IIoT) and Internet of Vehicles (IoV), it is often performed outside the database using frameworks such as Python or Spark. This "store first, compute later" workflow requires data to be exported for processing and then written back, introducing unnecessary data movement, higher latency, and greater system complexity—making real-time analytics increasingly difficult as data volumes grow.

A Different Approach: Compute Where the Data Lives

To eliminate unnecessary data movement, DolphinDB introduces the IoT Filtering module, bringing common filtering algorithms directly into the database.

Note: The FIR and IIR filtering functions require DolphinDB Server 3.00.6 or later.

Instead of relying on external processing frameworks, the module integrates multiple categories of filtering algorithms directly into the database engine, including:

Data Cleaning: Automatically remove sensor spikes (using median filtering), fill missing values, and eliminate outliers.
Signal Enhancement: Apply denoising algorithms such as Gaussian and Savitzky–Golay (SG) filters to extract signal trends that more accurately reflect the underlying system state.
Intelligent State Estimation: Integrate the Kalman filter to enable multi-sensor fusion and optimal state estimation.

Unlike traditional offline workflows, these algorithms can be embedded directly into DolphinDB's streaming engine, allowing sensor data to be cleaned and filtered before it is written to storage.

This architecture offers several advantages:

Reduce noisy data at the source
Eliminate unnecessary data transfer
Minimize ETL overhead
Enable low-latency online analytics
Simplify overall system architecture

Because storage, streaming, analytics, and filtering all run inside the same platform, developers no longer need to maintain multiple processing systems.

Practical Examples

1. Removing Sensor Spikes with Hampel Filtering

Industrial sensors occasionally produce impossible values—for example, a temperature reading suddenly jumping from 25°C to 1000°C due to interference.

These outliers can trigger false alarms or even unnecessary shutdowns.

The Hampel Filter provides a robust solution by detecting abnormal values using the median and Median Absolute Deviation (MAD) instead of the mean, making it resistant to extreme outliers.

// Simulated data with two obvious outliers
temp_raw = 25.0 + rand(1.0, 100)
temp_raw[10] = 100.0
temp_raw[50] = -50.0

// Hampel filtering
temp_clean =
    IotModules::Filtering::BasicStats::hampelFilter(
        temp_raw,
        window=5,
        k=3
    )

print("Outlier at 10: " + temp_raw[10] + " -> " + temp_clean[10])

Instead of manually identifying abnormal values, outliers are automatically replaced with the local median in real time.

2. Smoothing GPS Trajectories

GPS positioning often contains errors ranging from several meters to tens of meters, producing jagged trajectories.

Using Kalman filtering, DolphinDB combines noisy GPS observations with a motion model to continuously estimate the vehicle's true position.

This enables:

More accurate fleet tracking
Better dispatch decisions
Reduced empty mileage

3. Real-Time Streaming Data Cleaning

Industrial IoT systems continuously generate streaming sensor data.

DolphinDB allows filtering logic to be integrated directly into stream subscriptions so that every incoming message is cleaned immediately.

def cleanSensorData(msg){
    cleaned = IotModules::Filtering::BasicStats::hampelFilter(msg.temperature, 5, 2.0)
    t = table(msg.time as time, msg.deviceID as deviceID, msg.temperature as temperature, cleaned as cleaned)
    objByName("cleanResult").append!(t)
}

Each new sensor reading automatically triggers the predefined filtering logic, ensuring downstream applications always receive clean, high-quality data.

Performance Benchmark

Benchmark results show that DolphinDB significantly outperforms Python implementations across multiple filtering algorithms.

*Hampel Filter was tested on 1 million records, while Kalman Filter benchmarks used 1,000 observations due to the algorithm's sequential nature.

From Data Collection to Real-Time Decisions

Industrial IoT is no longer about who can collect the most data—it's about who can turn data into decisions the fastest.

By bringing data cleaning, denoising, sensor fusion, and state estimation directly into the database, DolphinDB eliminates unnecessary data movement and enables the entire processing pipeline to run in a single platform. Tasks that once depended on offline scripts and multiple systems can now be completed in milliseconds.

For industrial applications, that means lower latency, lower costs, and more reliable real-time decisions. Ultimately, the true value lies not in collecting more data, but in transforming it into actionable insights the moment it arriv

Get the complete code here.

Turning Noisy Sensor Data into Reliable Insights with In-Database Filtering