Standalone Mode
Configuration Parameters
In standalone mode, the configuration file is dolphindb.cfg. Configuration parameters can be specified either in dolphindb.cfg, or in the command line that starts DolphinDB.
Thread-related Parameters
There are the following types of threads in DolphinDB:
worker: threads that handle regular interactive tasks.
Since version 1.30.22/2.00.10 , DolphinDB has introduced a multilevel task assignment algorithm which divides the jobs and tasks (broken down from jobs) into different levels. The jobs and tasks are handled by the workers at the same levels. The levels of workers range from 0 to 5.
A job submitted to the node starts at level 0 and is handled by a level 0 worker. Based on the number of partitions involved in the job, the level 0 worker breaks it into an equivalent number of tasks. Local tasks are executed concurrently by the level 0 or level 1 workers, while remote tasks are sent to remote nodes by remote executors. Tasks sent to remote nodes are marked as level 1 and handled by level 1 workers there. If a task is further divided into subtasks, the remote subtasks are marked one level lower than their parent and handled by workers at the same level on the remote nodes.
With different levels of workers handling different levels of jobs and tasks, deadlock issues caused by circular dependencies across nodes can be effectively prevented.
Note: The number of workers determines the concurrent computing capability of a system.
remote executor: an independent thread that sends tasks to remote computing nodes. The remote executor is failure tolerant. If one machine fails, the remote executor will send the task to another machine with the data necessary for the task.
batch job worker: threads that handle batch jobs. Batch jobs are submitted with function submitJob and are usually time-consuming jobs. A batch job worker will be destroyed automatically after being idle for over 60 seconds.
dynamic worker: if there are jobs in the queue of regular interactive jobs and no jobs are completed in the previous second, the system will create dynamic workers to process the jobs in the queue. A dynamic worker will automatically be destroyed after being idle for 60 seconds. Dynamic workers make the system more responsive and can solve deadlocks among multiple nodes.
web worker: threads that handle HTTP requests. DolphinDB provides a web-based cluster management interface to interact with DolphinDB nodes.
urgent worker: threads that only handle time-sensitive system-level jobs such as logins (
login) and job cancellations (cancelJob/cancelConsoleJob). The configuration parameter urgentWorkerNum specifies the maximum number of urgent workers. The default is 1.
| Parameter | Description |
|---|---|
| WorkerNum | The size of worker pool for regular interactive jobs. The default value is the number of CPU cores. |
| remoteExecutors=1 | The number of remote executors. The default is 1. It is recommended to set it to the number of cluster nodes - 1. |
| maxBatchJobWorker=4 | The maximum number of batch job workers. The default value is the value of workerNum. |
| maxDynamicWorker=4 | The maximum number of dynamic workers. The default value is the value of workerNum. |
| WebWorkerNum=1 | The size of the web worker pool to process HTTP requests. The default value is 1. |
| urgentWorkerNum=1 | The maximum number of urgent workers. The default value is 1. |
| maxCachedBatchJobNum=2048 | The maximum number of batch jobs in the batch job queue. The default value is 2048. |
Memory-related Parameters
| Parameter | Description |
|---|---|
| chunkCacheEngineMemSize=0 | The capacity of cache engine in units of GB. After cache engine is enabled, data is not written to disk until data in cache exceeds 30% of chunkCacheEngineMemSize. The default value is 0indicating the cache engine is not enabled. To enable the cache engine, we must set chunkCacheEngineMemSize>0 and dataSync=1. |
| maxMemSize=0 | The maximum memory (in GB) allocated to DolphinDB. If set to 0, it means no limits on memory usage. |
| memoryReleaseRate=5 | The rate at which unused memory is released to the operating system. It is a floating point number between 0 and 10. memoryReleaseRate=0 means that unused memory will not be releasedactively; memoryReleaseRate=10 means that memory will be released at the fastest speed. The default value is 5. |
| regularArrayMemoryLimit=256 | The limit on the memory size (in units of MB) of a regular array. Must be a power of 2. The default value is 512. |
| warningMemSize | When memory usage exceeds warningMemSize (in units of GB) , the system will automatically clean up the cache of some databases to avoid OOM exceptions. The default value is 75% ofMaxMemSize. |
| reservedMemSize | A positive number specified in units of GB. When the available memory in DolphinDB is less than reservedMemSize, DolphinDB only allocates a memory block of limited size (as specified by maxBlockSizeForReservedMemory) for each memory request. reservedMemSize is provided to restrict the memory allocation for each memory request to improve the likelihood that there is enough memory for critical operations that use a small amount of memory (error reporting, rollbacks, etc.). For example, when data writes fail due to insufficient memory, the transactions can be rolled back to guarantee data consistency. If the parameter is not specified, the system sets reservedMemSize = 5% * maxMemSize and reservedMemSize must be between 64MB and 1GB. |
| maxBlockSizeForReservedMemory | The maximum size (in units of KB) of the memory block that DolphinDB allocates for each memory request when its available memory is less than reservedMemSize. The default value is 64. It is not recommended to set it too high as exceptions or crashes may occur if there isn’t enough memory left for critical database operations. |
Hard Disk Related Parameters
To mitigate I/O bottlenecks, a data node can be configured with multiple volumes to improve the speed of parallel read and write.
Log files include metadata log file, redo log, query log, system log and batch job log. They have different requirements for disk performance. Generally, metadata log and redo log should be stored on SSDs while other logs can be stored on HDDs.
| Parameter | Description |
|---|---|
| batchJobDir | The folder for batch job logs and results. The default value is <HomeDir>/batchJobs. |
| chunkMetaDir | The folder for the metadata of data chunks on each data node. The default value is <HomeDir> /storage/CHUNK_METADATA. In a cluster, each data node should be configuredwith different chunkMetaDir. |
| dataSync=0 | If dataSync=1, DolphinDB generates redo log when the database is being updated. It can avoid data loss in the event of a system crash or power outage. The default value ofdataSync is 0. |
| jobLogFile=jobLog.csv | The path and name of the job log file that contains descriptive information of all the queries that have been executed for each node. It must be a csv file. The default folder for thejob log file is the log folder. The default name of the job log file is nodeAlias_job.log. |
| logFile=DolphinDBlog | The path and name of the log file. It displays the server configuration specifications, warnings and error messages. |
| logLevel=INFO | The log file only keeps log information equal to higher than the specified level. From the lowest to the highest level, the possible values are DEBUG, INFO, WARNING andERROR. The default value is INFO. |
| maxLogSize=1024 | The system will archive the server log after the log reaches the specified size limit (in MB). The default value is 1024 and the minimum value is 100. The archived log filename adds a prefix to the original log file name. The prefix is in the format of \date\seq, e.g. 20181109000. seq has 3 digits and starts with 000 |
| redoLogDir | The directory of the redo log. The default value is /log/redoLog. |
| redoLogPurgeInterval=30 | Clear the redo log of transactions whose data have been persisted at intervals specified by redoLogPurgeInterval in terms of seconds. The default value is 30. |
| redoLogPurgeLimit=4000 | Clear the redo log of transactions whose data have been persisted if the size of the redo log exceeds redoLogPurgeLimit in terms of MB. The default value is 4000. |
| volumes=/hdd/hdd1/volumes,/hdd/hdd2/volumes,/hdd/hdd3/volumes,/hdd/hdd4/volumes | The folder where data files are saved in the distributed file system on a data node. The default value is <HomeDir>/<nodeAlias>/storage. |
| diskIOConcurrencyLevel=4 | The number of threads for reading and writing to disk. The default value is 1. diskIOConcurrencyLevel = 0 means the threads handling the relevant tasks will read and write to the disk. When diskIOConcurrencyLevel > 0, the system will create the specified number of threads to read and write to the disk concurrently. For HDDs, it is recommended to set diskIOConcurrencyLevel to the number of volumes configured on the node (specified by the configuration parameter volumes). For SSDs, it is recommended to set diskIOConcurrencyLevel = 0 |
| marketHolidayDir | Specifies the directory to the file of market holidays. Either absolute or relative directory can be specified. The system searches the relative directory in the following order: home directory of the node, the working directory of the node, and the directory with the DolphinDB executable. The default directory is <HomeDir> /marketHoliday. The file must be a single-column CSV file with DATE data. Based on the CSV files under marketHolidayDir, a trading calendar is created with the file name as its identifier. Note that: (1) Weekends are recognized as holidays by default, therefore, only weekday holidays need to be filled in the file. (2) It is recommended to name the file in Market Identifier Code, e.g., "XNYS.csv". |
Login-related Parameter
| Parameter | Description |
|---|---|
| thirdPartyAuthenticator | Specifies a function view (which takes the same arguments as login) for third-party authentication. Users can connect to a third-party system through DolphinDB plugins such as HttpClient. Once specified, DolphinDB server authenticates the user login through the third-party system. |
Network-related Parameters
| Parameter | Description |
|---|---|
| enableHTTPS=false | Whether to enable HTTPS protocol. The default value is false. |
| localSite=localhost:8848:local8848 | Host address, port number and alias of the local node, separated by ":". In standalone model, the default value is localhost:8848:local8848. |
| maxConnections=64 | The maximum number of connections (from GUI, API, other nodes, etc) to the local node. |
| maxConnectionPerSite=2 | The maximum number of remote nodes that can be connected to from the local node. |
| tcpNoDelay=1 | Whether to enable the TCP_NODELAY socket option. The default value is false. |
From DolphinDB 1.30.21/2.00.9 onwards, license server is supported for resource allocation. Note that license server is only available with license of type 3. You may contact our technical support for more information.
| Parameter | Description |
|---|---|
| licenseServerSite | Specifies the LAN information of the node where the license server is located. It is specified in the format of <host:port:aliasgt;, where port must be the same as that specified in the license file. The license server is a standalone server that assigns hardware resources to each node (except the agent) in a cluster. If licenseServerSite is configured with the same value as localSite, license server runs on the current node; Otherwise the current node requests resources from the license server. If licenseServerSite is not configured, the current node uses the local license file. |
| bindCores | Specifies the CPU cores (by core ID) for the node. To specify multiple CPU cores, you can separate the core IDs with a comma (","); To specify consecutive cores, you can use a hyphen ("-") to join the starting and ending IDs. For example, to bind CPU cores numbered 1 to 4, it can be 1, 2, 3, 4 or 1-4. Note that this parameter only takes effect on the node that requests resources from the license server. If it is configured, the process runs on the specified cores. If it is not configured, the process runs on fixed cores according to the license file; Otherwise random CPU cores are used. |
Streaming-related Parameters
The required configuration parameters for a publisher node are maxPubConnections and persistenceDir.
| Parameter | Description |
|---|---|
| maxMsgNumPerBlock=1024 | The maximum number of records in a message block. The default value is 1024. |
| maxPersistenceQueueDepth=10000000 | The maximum depth (number of records) of a message queue to persist a stream table to disk. The default value is 10,000,000. |
| maxPubQueueDepthPerSite=10000000 | The maximum depth (number of records) of a message queue on the publisher node. The default value is 10,000,000. |
| maxPubConnections=0 | The maximum number of subscriber nodes that the publisher node can connect to. The default value is 0. For the node to server as a publisher, we must set maxPubConnections>0. |
| persistenceDir=/home/DolphinDB/Data/Persistence | The directory where shared streaming tables are persisted to. To enable persistence, persistenceDir must be specified. In a cluster, each data node should be configured with different persistenceDir. |
| persistenceWorkerNum=1 | The number of workers responsible for persisting streaming tables to disk in asynchronous mode. The default value is 0. |
| Parameter | Description |
|---|---|
| MaxSubConnections=64 | The maximum number of publishers that the subscriber node can connect to. The default value is 64. |
| MaxSubQueueDepth=10000000 | The maximum depth (number of records) of a message queue on the subscriber node. |
| persistOffsetDir=/home/DolphinDB/streamlog | The directory to save the offset of the last subscribed message that has been processed. If persistOffsetDir is not specified, it will be saved under the directory specified by persistenceDir. If persistenceDir isnot specified, it will be saved in the streamLog folder under the home directory. |
| subExecutorPooling=true | A Boolean value indicating whether streaming executors use pooling mode. The default value is false. If it is set to 0, it means the thread can conduct message parsing and can also process messages. |
| subExecutors=2 | The number of message processing threads in the subscriber node. Only when subscription is enabled is this parameter relevant. The default value is 1. If it is set to 0, it means the thread can conductmessage parsing and can also process messages. |
| subPort=8000 | The port number that the subscription thread is listening on. This parameter must be specified to enable the node(s) to serve as subscriber(s). |
| subThrottle=1000 | A non-negative integer in milliseconds, indicating the interval at which the system checks whether throttle in function subscribeTable has been reached. The default value is 1000. If the interval specified by the parameter throttle in subscribeTable is less than the configuration parameter subThrottle, the parameter handler in subscribeTable will be triggered to process messages at an interval of subThrottle. To set If throttle is less than 1 second, you need to modify the configuration parameter subThrottle first. For example, to set throttle =0.001s (second), please set subThrottle =1 first. Note: This parameter is only valid if the parameter batchSize is specified in function subscribeTable. |
System Management Parameters
| Parameter | Description |
|---|---|
| console=1 | The configuration file. The default value is dolphindb.cfg. It can only be specified in command line. |
| config=dolphindb.cfg | A Boolean value indicating whether to start a DolphinDB console. The default value is 0. It can only be specified in command line. |
| home=/home/DolphinDB/server | The DolphinDB home directory where the configuration files, the license file, the log file and other related files are located. It can only be specified in command line. |
| logFile=dolphindb.log | The log file. The default value is dolphindb.log. It can only be specified in command line. |
| maxPartitionNumPerQuery=65536 | The maximum number of partitions that a single query can search. The default value is 65536. |
| mode=single | single: standalone mode; datanode: data node; controller: controller node; agent: agent node. |
| moduleDir=modules | The directory for the module files. The default value is the relative directory "modules" that needs to be created by the user. The system searches the relative directory "modules" in the following order: home directory of the node, the working directory of the node, and the directory with the DolphinDB executable. |
| newValuePartitionPolicy=skip | Specifies how new data outside of existing VALUE partitions are handled in databases. It can be "add", "skip", or "fail". - skip (default): appends data that belongs to existing VALUE partitions and ignores out-of-scope data. - add: Creates new VALUE partitions to add all data. - fail: Throws exception if out-of-scope data is included. None of the data will be written. In most cases, add is recommended. |
| allowMissingPartitions=true | Specifies the behavior when incoming data contains new partition values that do not match any existing partitions. true (default) means to write the data that belongs to existing partitions and ignore the out-of-scope data. False means not to write any incoming data to the database and throw an exception. Note: For VALUE partitions where newValuePartitionPolicy is set to “add“ or “fail“, data will be handled based on the corresponding rules. In those cases, allowMissingPartitions does not take effect. |
| perfMonitoring=1 | Whether to enable performance monitoring. The default value is false for the standalone mode and true for the cluster mode. |
| pluginDir=plugins | The directory for the plugin files. The default value is the relative directory "plugins". The system searches the relative directory "plugins" in the following order: home directory of the node, the working directoryof the node, and the directory with the DolphinDB executable. |
| preloadModules=plugins::mysql, system::log::fileLog | The modules or plugins that are loaded after the system starts. Use commas to separate multiple modules/plugins. |
| init=dolphindb.dos | This file is executed when the system starts. The default file is <HomeDir> /dolphindb.dos. It usually contains definitions of system-level functions that are visible to all users and cannot be overwritten. |
| startup=startup.dos | This file is executed after the system starts. The default file is <HomeDir> /startup.dos. It can be used to load plugins, load tables and share them, define and load stream tables, etc. |
| postStart=postStart.dos | This file is executed after scheduled jobs are initialized to load scheduled jobs. The default file is <HomeDir> /postStart.dos. |
| run=dailyJobs.dos | This file is executed after the startup script (startup.dos) is executed. The default folder is DolphinDB home directory. It can be used to schedule jobs. After this file is executed, the system terminates. |
| tzdb=C:DolphinDBservertzdb | The directory of the time zone database. The default value is homeservertzdb. |
| webRoot=/home/DolphinDB/server/web | The directory of the web server. The default value is <HomeDir> /web. |
| webLoginRequired=false | Whether a user must log in to use the web-based cluster manager. The default value is false. |
| useHardLink=true | Whether to use the hardlink feature of the file system. Set to true to use the system hardlink feature; Set to false to not use the hardlink feature. The default value is true. |
| maxFileHandles | Sets the maximum number of file descriptors open in a DolphinDB process. |
| enableConcurrentDimensionalTableWrite=false | Whether to allow conducting concurrent write or update/delete on dimension tables. The default value is false, indicating concurrent write and update/delete is disabled for dimension tables. |
| removeSpecialCharInColumnName =false | Whether to allow column names to contain special characters or to start without a letter. The default value is false, which can be used to remove the special characters in the column names generated by "pivot by". Specify the value as true to be compatible with version 1.30.14 and before. |
| nullAsMinValueForComparison=t rue | Whether a NULL value is treated as the minimum value in data comparison. The default value is true. If it is set to "false", the result of comparison involving NULL value is NULL. |
| enableChunkGranularityConfig= false | Specify the chunk granularity to determine the level of the lock of a DolphinDB transaction. When writing to a chunk, the transaction locks it to prevent other transactions from writing to it. Before version 1.30.16/2.00.4, the chunk granularity is set at the database level, i.e., each partition in the database is a chunk. In this case, concurrent writes to different tablets in the same partition are not allowed. This configuration parameter was released with version 1.30.16/2.00.4. The default value is false, which indicates the chunk granularity is at the table level, i.e., each tablet of a partition is a chunk. Concurrent writes to different tables in the same partition are thus allowed. If it is set to true, you can specify the chunk granularity with the parameter chunkGranularity of function database. |
| logRetentionTime=30 | The amount of time to keep a log file before deleting it. The default value is 30 days (of floating-point data type). For example, 0.5 means 12 hours. If set to 0, log files are not deleted. |
| oldChunkVersionRetentionTime=60 | Sets the retention time (in minutes) for the old chunk versions. The upper limit is 240. When SQL update/upsert/delete is executed, the system generates a new replica of the chunk (named by "table_name_tid") to perform the update and delete operations. Each old chunk will be retained in the system for the specified retention time, and a maximum of 5 old chunks can be kept at the same time. |
Core Dumps
| Parameter | Description |
|---|---|
| enableCoreDump=true | [Linux only] Whether to check and enable core dumps when starting DolphinDB. The default value is true. True means that DolphinDB will check if core dumps are enabled during startup. If core dumps are disabled, it will try to enable them, provided the system's hard limit allows core dumps of a sufficient size. If core dumps are already enabled, no changes will be made. False means that DolphinDB will start up without checking or modifying the system's current core dump enablement state. |
| disableCoreDumpOnShutdown=false | [Linux only] Whether to disable core dumps during a graceful shutdown. The default value is false, meaning core dumps will not be disabled during a graceful shutdown. This parameter only takes effect for the local node. |
Parameters for DolphinDB terminal
DolphinDB terminal is a command line interactive tool for connecting to a remote DolphinDB server to execute commands. It is supported in 1.10.4 and above.
Start the DolphinDB terminal in the Linux system with the following command:
rlwrap -r ./dolphindb -remoteHost 192.168.1.135 -remotePort 8848
Start the DolphinDB terminal in Windows with the following command:
dolphindb.exe -remoteHost 192.168.1.135 -remotePort 8848
After starting up successfully, the following is displayed:
DolphinDB Terminal 1.10.4 (Build:2020.04.03). Copyright (c) 2011~2020 DolphinDB, Inc.
To quit the terminal:
quit
The following are the parameters related to the DolphinDB terminal. Note that these parameters can only be used on the command line.
| Parameter | Description |
|---|---|
| remoteHost | IP address of the remote DolphinDB server. |
| remotePort | Port number of the remote DolphinDB server. |
| uid | User name of the remote DolphinDB server. |
| pwd | Password of the remote DolphinDB server. |
| run | The local DolphinDB script file that is sent to the remote DolphinDB server for execution when starting up the terminal. After the execution is completed, the terminal will automatically exit. |
Note: If an exception occurs when running a local script with -run on a remote server, the exception return value is 255.