Example 3. Compress data




To save hard disk space, we may want to save tables in compressed format. DolphinDB uses LZ4 algorithm to compress data. LZ4 has excellent compression/decompression speed and decent compression ratio.


In the following example, table t is saved on disk as an uncompressed table t1, and as a compressed table t3.



db=database("C:/DolphinDB/Data/db1")

n=10000000

t=table(take(1..10000, n).sort() as id, take(2000.12.31+1..1000, n) as date, rand(10, n) as x, rand(10.0, n) as y)

saveTable(db, t, `t1)

saveTable(db, t, `t3, 0, 1);        // appending mode is off; compression mode is on.



We can compare the size of each column before and after compression.




If we save the file "USstocks" in compressed format, it reduces the size of the file from 3.56GB to 897MB.



USstocks = ploadText("c:/DolphinDB/Data/USstocks.csv")

saveTable(db, USstocks, `USstocks1)

saveTable(db, USstocks, `USstocks2, 0, 1);        // appending mode is off; compression mode is on.



Download source code here.