Standalone Mode

Configuration Parameters

In standalone mode, the configuration file is dolphindb.cfg. Configuration parameters can be specified either in dolphindb.cfg, or in the command line that starts DolphinDB.

Thread-related Parameters

There are 6 types of threads in DolphinDB:

  • worker: threads that handle regular interactive tasks. A worker can execute the requested job. It can also break the job into sub-tasks to be executed by local executors and/or remote executors.

  • local executor: threads that execute sub-tasks on local machines assigned by workers. Each local executor can only process one sub-task at a time. Local executors are shared by all workers.

    More workers and local executors can speed up parallel computing.

  • remote executor: an independent thread that sends sub-tasks to remote computing nodes. Each node has one and only one remote executor. The remote executor is failure tolerant. If one machine fails, the remote executor will send the task to another machine with the data necessary for the task.

  • batch job worker: threads that handle batch jobs. Batch jobs are submitted with function submitJob and are usually time-consuming jobs. A batch job worker will be destroyed automatically after being idle for over 60 seconds.

  • dynamic worker: if there are jobs in the queue of regular interactive jobs and no jobs are completed in the previous second, the system will create dynamic workers to process the jobs in the queue. A dynamic worker will automatically be destroyed after being idle for 60 seconds. Dynamic workers make the system more responsive and can solve deadlocks among multiple nodes.

  • web worker: threads that handle HTTP requests. DolphinDB provides a web-based cluster management interface to interact with DolphinDB nodes.

Configuration Parameters

Details

localExecutors=3

The number of local executors. The default value is the number of CPU cores - 1.

maxBatchJobWorker=4

The maximum number of batch job workers. The default value is the value of workerNum.

maxDynamicWorker=4

The maximum number of dynamic workers. The default value is the value of workerNum.

WebWorkerNum=1

The size of the web worker pool to process HTTP requests. The default value is 1.

WorkerNum=4

The size of worker pool for regular interactive jobs. The default value is the number of CPU cores.

maxCachedBatchJobNum=2048

The maximum number of batch jobs in the batch job queue. The default value is 2048.

Memory-related Parameters

Configuration Parameters

Details

chunkCacheEngineMemSize=0

The capacity of cache engine in units of GB. After cache engine is enabled, data is not written to disk until data in cache exceeds 30% of

chunkCacheEngineMemSize. The default value is 0 indicating the cache engine is not enabled. To enable the cache engine, we must set chunkCacheEngineMemSize > 0

and dataSync = 1.

maxMemSize=0

The maximum memory (in units of GB) allocated to DolphinDB. If set to 0, it means no limits on memory usage.

memoryReleaseRate=5

The rate at which unused memory is released to the operating system. It is a floating point number between 0 and 10. memoryReleaseRate = 0 means that unused

memory will not be released actively; memoryReleaseRate = 10 means that memory will be released at the fastest speed. The default value is 5.

regularArrayMemoryLimit=2 048

The limit on the memory size (in units of MB) of a regular array. Must be a power of 2. The maximum value allowed is 2048 MB, which is also the default value.

Configurations exceeding 2048 will not take effect.

warningMemSize

When memory usage exceeds warningMemSize (in units of GB), the system will automatically clean up the cache of some databases to avoid OOM exceptions. The

default value is 75% of MaxMemSize.

reservedMemSize

A positive number specified in units of GB. When the available memory in DolphinDB is less than reservedMemSize, DolphinDB only allocates a memory block of

limited size (as specified by maxBlockSizeForReservedMemory) for each memory request. reservedMemSize is provided to restrict the memory allocation for each

memory request to improve the likelihood that there is enough memory for critical operations that use a small amount of memory (error reporting, rollbacks, etc.).

For example, when data writes fail due to insufficient memory, the transactions can be rolled back to guarantee data consistency. If the parameter is not

specified, the system sets reservedMemSize = 5% * maxMemSize and reservedMemSize must be between 64MB and 1GB.

maxBlockSizeForReservedMemory

The maximum size (in units of KB) of the memory block that DolphinDB allocates for each memory request when its available memory is less than reservedMemSize.

The default value is 64. It is not recommended to set it too high as exceptions or crashes may occur if there isn’t enough memory left for critical database

operations.

Hard Disk Related Parameters

To mitigate I/O bottlenecks, a data node can be configured with multiple volumes to improve the speed of parallel read and write.

Log files include metadata log file, redo log, query log, system log and batch job log. They have different requirements for disk performance. Generally, metadata log and redo log should be stored on SSDs while other logs can be stored on HDDs.

Configuration Parameters

Details

batchJobDir

The folder for batch job logs and results. The default value is <HomeDir>/batchJobs.

chunkMetaDir

The folder for the metadata of data chunks on each data node. The default value is <HomeDir>/storage/CHUNK_METADATA. In a cluster, each data node should be

configured with different chunkMetaDir.

dataSync=0

If dataSync=1, DolphinDB generates redo log when the database is being updated. It can avoid data loss in the event of a system crash or power outage. The default

value of dataSync is 0.

jobLogFile=jobLog.csv

The path and name of the job log file that contains descriptive information of all the queries that have been executed for each node. It must be a csv file. The

default folder for the job log file is the log folder. The default name of the job log file is nodeAlias_job.log.

logFile=DolphinDBlog

The path and name of the log file. It displays the server configuration specifications, warnings and error messages.

logLevel=INFO

The log file only keeps log information equal to higher than the specified level. From the lowest to the highest level, the possible values are DEBUG, INFO,

WARNING and ERROR. The default value is INFO.

maxLogSize=1024

The system will archive the server log after the log reaches the specified size limit (in units of MB). The default value is 1024 and the minimum value is 100. The

archived log file name adds a prefix to the original log file name. The prefix is in the format of <date><seq>, e.g. seq 20181109000 has 3 digits and starts

with “000”.

redoLogDir

The directory of the redo log. The default value is /log/redoLog.

redoLogPurgeInterval=30

Clear the redo log of transactions whose data have been persisted at intervals specified by redoLogPurgeInterval in terms of seconds. The default value is 30.

redoLogPurgeLimit=4

The maximum disk space (in GB) for the redo log. The default value is 4. If the size of redo log exceeds redoLogPurgeLimit, the system will automatically purge

the redo log.

volumes=/hdd/hdd1/volumes,/hd

d/hdd2/volumes,/hdd/hdd3/volu

mes,/hdd/hdd4/volumes

The folder where data files are saved in the distributed file system on a data node. The default value is <HomeDir>/<nodeAlias>/storage.

diskIOConcurrencyLevel=1

The number of threads for reading and writing to disk. The default is 1.

diskIOConcurrencyLevel = 0 means the threads handling the relevant tasks will read and write to the disk. When diskIOConcurrencyLevel > 0, the system will create

the specified number of threads to read and write to the disk concurrently.

For HDDs, it is recommended to set diskIOConcurrencyLevel to the number of volumes configured on the node (specified by the configuration parameter volumes).

For SSDs, it is recommended to set diskIOConcurrencyLevel = 0

Network-related Parameters

Configuration Parameters

Details

enableHTTPS=false

Whether to enable HTTPS protocol. The default value is false.

localSite=localhost:8848:local

8848

Host address, port number and alias of the local node, separated by “:”. In standalone model, the default value is localhost:8848:local8848.

maxConnections=64

The maximum number of connections (from GUI, API, other nodes, etc) to the local node.

maxConnectionPerSite=2

On windows, the default value is 64 as well as the effective maximum value; On Linux, the default value is 512.

tcpNoDelay=0

Whether to enable the TCP_NODELAY socket option. The default value is 0.

Streaming-related Parameters

The required configuration parameters for a publisher node are maxPubConnections and persistenceDir.

Configuration Parameters

Details

maxMsgNumPerBlock=1024

The maximum number of records in a message block. The default value is 1024.

maxPersistenceQueueDepth=10000000

The maximum depth (number of records) of a message queue to persist a stream table to disk. The default value is 10,000,000.

maxPubQueueDepthPerSite=10000000

The maximum depth (number of records) of a message queue on the publisher node. The default value is 10,000,000.

maxPubConnections=0

The maximum number of subscriber nodes that the publisher node can connect to. The default value is 0. For the node to server as a publisher, we must set maxPubConnections > 0.

persistenceDir=/home/DolphinDB/

Data/Persistence

The directory where shared streaming tables are persisted to. To enable persistence, persistenceDir must be specified. In a cluster, each data node should be

configured with different persistenceDir.

persistenceWorkerNum=1

The number of workers responsible for persisting streaming tables to disk in asynchronous mode. The default value is 0.

The required subscriber parameters for a publisher node is subPort.

Configuration Parameters

Details

MaxSubConnections=64

The maximum number of publishers that the subscriber node can connec to. The default value is 64.

MaxSubQueueDepth=10000000

The maximum depth (number of records) of a message queue on the subscriber node.

persistOffsetDir=/home/DolphinDB/

streamlog

The directory to save the offset of the last subscribed message that has been processed. If persistOffsetDir is not specified, it will be saved under the directory

specified by persistenceDir. If persistenceDir is not specified, it will be saved in the streamLog folder under the home directory.

subExecutorPooling=true

A Boolean value indicating whether streaming executors use pooling mode. The default value is false.

subExecutors=2

The number of message processing threads in the subscriber node. Only when subscription is enabled is this parameter relevant. The default value is 1. If it is set

to 0, it means the thread can conduct message parsing and can also process messages.

subPort=8000

The port number that the subscription thread is listening on. This parameter must be specified to enable the node(s) to serve as subscriber(s).

subThrottle=1000

A non-negative integer in milliseconds, indicating the interval at which the system checks whether throttle in function subscribeTable has been reached. The

default value is 1000. If the interval specified by the parameter throttle in subscribeTable is less than the configuration parameter subThrottle, the

parameter handler in subscribeTable will be triggered to process messages at an interval of subThrottle. To set If throttle is less than 1 second, you

need to modify the configuration parameter subThrottle first. For example, to set throttle =0.001s (second), please set subThrottle =1 first. (Note: This

parameter is only valid if the parameter batchSize is specified in function subscribeTable.)

System Management Parameters

Configuration Parameters

Details

console=1

The configuration file. The default value is dolphindb.cfg. It can only be specified in command line.

config=dolphindb.cfg

A Boolean value indicating whether to start a DolphinDB console. The default value is 0. It can only be specified in command line.

home=/home/DolphinDB/server

The DolphinDB home directory where the configuration files, the license file, the log file and other related files are located. It can only be specified in command

line.

logFile=dolphindb.log

The log file. The default value is dolphindb.log. It can only be specified in command line.

maxPartitionNumPerQuery=65536

The maximum number of partitions that a single query can search. The default value is 65536.

mode=single

single: standalone mode; datanode: data node; controller: controller node; agent: agent node.

moduleDir=modules

The directory for the module files. The default value is the relative directory “modules” that needs to be created by the user. The system searches the relative

directory “modules” in the following order: home directory of the node, the working directory of the node, and the directory with the DolphinDB executable.

newValuePartitionPolicy=add

How the system deals with new data that is outside the partition scheme for a VALUE domain (or a VALUE domain in a COMPO domain). If set to “skip”, new data is not

saved and no exception is thrown; if set to “fail”, new data is not saved and an exception is thrown; if set to “add”, new partitions are created to save the new

data. The default value is “add”.

perfMonitoring=1

Whether to enable performance monitoring. The default value is false for the standalone mode and true for the cluster mode.

pluginDir=plugins

The directory for the plugin files. The default value is the relative directory “plugins”. The system searches the relative directory “plugins” in the following

order: home directory of the node, the working directory of the node, and the directory with the DolphinDB executable.

preloadModules=plugins::mysql

, system::log::fileLog

The modules or plugins that are loaded after the system starts. Use commas to separate multiple modules/plugins.

init=dolphindb.dos

This file is executed when the system starts. The default file is <HomeDir>/dolphindb.dos. It usually contains definitions of system-level functions that are

visible to all users and cannot be overwritten.

startup=startup.dos

This file is executed after the system starts. The default file is <HomeDir>/startup.dos. It can be used to load plugins, load tables and share them, define and

load stream tables, etc.

postStart=postStart.dos

This file is executed after scheduled jobs are initialized to load scheduled jobs. The default file is <HomeDir>/postStart.dos.

run=dailyJobs.dos

This file is executed after the startup script (startup.dos) is executed. The default folder is DolphinDB home directory. It can be used to schedule jobs. After

this file is executed, the system terminates.

tzdb=/home/DolphinDB/server/t

zdb

The directory of the time zone database. The default value is <HomeDir>/server/tzdb.

webRoot=/home/DolphinDB/server

/web

The directory of the web server. The default value is <HomeDir>/web.

useHardLink=true

Whether to use the hardlink feature of the file system. Set to true to use the system hardlink feature; Set to false to not use the hardlink feature. The default

value is true.

enableConcurrentDimensionalTa

bleWrite=false

Whether to allow conducting concurrent write or update/delete on dimension tables. The default value is false, indicating concurrent write and update/delete is

disabled for dimension tables.

removeSpecialCharInColumnName

=false

Whether to allow column names to contain special characters or to start without a letter. The default value is false, which can be used to remove the special

characters in the column names generated by “pivot by”. Specify the value as true to be compatible with version 1.30.14 and before.

nullAsMinValueForComparison=t

rue

Whether a NULL value is treated as the minimum value in data comparison. The default value is true. If it is set to “false”, the result of comparison involving

NULL value is NULL.

enableChunkGranularityConfig=

false

Specify the chunk granularity to determine the level of the lock of a DolphinDB transaction. When writing to a chunk, the transaction locks it to prevent other

transactions from writing to it. Before version 1.30.16/2.00.4, the chunk granularity is set at the database level, i.e., each partition in the database is a

chunk. In this case, concurrent writes to different tablets in the same partition are not allowed. This configuration parameter was released with version

1.30.16/2.00.4. The default value is false, which indicates the chunk granularity is at the table level, i.e., each tablet of a partition is a chunk. Concurrent

writes to different tables in the same partition are thus allowed. If it is set to true, you can specify the chunk granularity with the parameter

chunkGranularity of function database.

Parameters for DolphinDB terminal

DolphinDB terminal is a command line interactive tool for connecting to a remote DolphinDB server to execute commands. It is supported in 1.10.4 and above.

Start the DolphinDB terminal in the Linux system with the following command:

$ rlwrap -r ./dolphindb -remoteHost 192.168.1.135 -remotePort 8848

Start the DolphinDB terminal in Windows with the following command:

$ dolphindb.exe -remoteHost 192.168.1.135 -remotePort 8848

After starting up successfully, the following is displayed:

$ DolphinDB Terminal 1.10.4 (Build:2020.04.03). Copyright (c) 2011~2020 DolphinDB, Inc.

To quit the terminal:

$ quit

The following are the parameters related to the DolphinDB terminal. Note that these parameters can only be used on the command line.

Configuration Parameters

Details

remoteHost

IP address of the remote DolphinDB server.

remotePort

Port number of the remote DolphinDB server.

stdoutLog

Where to output the system log. The default value is 0 indicating file dolphindb.log; 1 means stdout; 2 means both stdout and file dolphindb.log.

uid

User name of the remote DolphinDB server.

pwd

Password of the remote DolphinDB server.

run

The local DolphinDB script file that is sent to the remote DolphinDB server for execution when starting up the terminal. After the execution is completed, the terminal will automatically exit.