Add Partitions

DolphinDB supports adding new partitions for databases with VALUE or RANGE domain, or databases with VALUE or RANGE domain in COMPO domain.

Add Partitions for VALUE Domain

We can add new partitions for a partitioned database with VALUE domain:

Set the configuration parameter newValuePartitionPolicy=add. In cluster mode, it is set in cluster.cfg. In standalone mode m it is set in dolphindb.cfg.
Use function addValuePartitions .

For a partitioned database with VALUE domain, if the new data does not belong to the existing partitioning scheme, we can specify how the new data is processed with the configuration parameter newValuePartitionPolicy.

1. By default, newValuePartitionPolicy=skip. The records in the new data that belong to the existing partitioning scheme will be saved to the database, while those that don’t belong to the existing partitioning scheme will not be saved to the database.

2. newValuePartitionPolicy=fail. If the new data has at least one records that doesn’t belong to the existing partitioning scheme, an exception is thrown and none of the new data will be saved to the database.

3. newValuePartitionPolicy=add. If the new data has records that doesn’t belong to the existing partitioning scheme, new partitions will be automatically generated. All of the new data will be saved to the database.

If newValuePartitionPolicy=skip or fail, we can use function addValuePartitions to extend partitioning scheme. In the following example, we add new date partitions from 2017.08.12 to 2017.08.20 in database “dfs://compoDB”.

$ n=1000000
$ ID=rand(100, n)
$ dates=2017.08.07..2017.08.11
$ date=rand(dates, n)
$ x=rand(10.0, n)
$ t=table(ID, date, x);

$ dbID=database(, RANGE, 0 50 100);
$ dbDate = database(, VALUE, 2017.08.07..2017.08.11)
$ db = database("dfs://compoDB", COMPO, [dbID, dbDate]);
$ pt = db.createPartitionedTable(t, `pt, `ID`date)
$ t.append!(t);

$ addValuePartitions(db,2017.08.12..2017.08.20,1);

9

Please note: the database should be reloaded after adding new partitions.

$ db=database("dfs://compoDB")
$ pt=loadTable(db,"pt")
$ t1=table(0..99 as ID,take(2017.08.12,100) as date,rand(10.0,100) as x)
$ pt.append!(t1)
$ select count(*) from loadTable("dfs://compoDB","pt");

1000100

Add Partitions for RANGE Domain

For a partitioned database with RANGE domain, new partitions cannot be automatically generated with a configuration parameter. Instead, we can use function addRangePartitions to extend the existing partitioning scheme. To add 3 new ID partitions [100,150), [150,200) and [200,250) to the example above:

$ addRangePartitions(db,100 150 200 250,0);

3

The database should be reloaded after adding new partitions.

$ db=database("dfs://compoDB")
$ pt=loadTable(db,"pt")
$ t1=table(rand(100..249,10000) as ID,rand(2017.08.07..2017.08.12,10000) as date,rand(10.0,10000) as x)
$ pt.append!(t1)
$ select count(*) from loadTable("dfs://compoDB","pt");

1010100