Edit

Deploy Broker

This topic describes how to deploy Broker. With Broker, StarRocks can read data from source such as HDFS and S3, and pre-process, load, and backup the data with its own computing resources.

Download and decompress the installer

Download and decompress StarRocks installer.

tar -xzvf StarRocks-x.x.x.tar.gz

CAUTION

Replace the file name in the command as the real file name you downloaded.

Configure a Broker

Navigate to StarRocks-x.x.x/apache_hdfs_broker.

cd StarRocks-x.x.x/apache_hdfs_broker

CAUTION

Replace the file name in the command as the real file name you downloaded.

Specify the Broker configuration file conf/apache_hdfs_broker.conf. Because the default configuration can be used directly, no configuration item is changed in the following example. You can copy your own HDFS cluster configuration file and paste it under conf directory.

Start a Broker

Run the following command to start Broker.

./apache_hdfs_broker/bin/start_broker.sh --daemon

Add a Broker to cluster

Run the following command to add the Broker to cluster.

ALTER SYSTEM ADD BROKER broker_name "broker_ip:broker_port";

NOTE

  • By default, the port for Broker is 8000.
  • When multiple Brokers are added to a cluster at the same time, they share the same broker_name.

verify if Broker starts

You can verify if the BE node is started properly via MySQL client.

SHOW PROC "/brokers"\G

Example:

MySQL [(none)]> SHOW PROC "/brokers"\G
*************************** 1. row ***************************
          Name: broker1
            IP: x.x.x.x
          Port: 8000
         Alive: true
 LastStartTime: 2022-05-19 11:21:36
LastUpdateTime: 2022-05-19 11:28:31
        ErrMsg:
1 row in set (0.00 sec)

When the field Alive is true, the Broker is properly started and added to the cluster.

Stop a Broker

Run the following command to stop a Broker.

./bin/stop_broker.sh --daemon

Configure a broker

You can only set the configuration items of a broker by changing them in the corresponding configuration file broker.conf, and restart the broker to allow the change to take effect.

Configuration itemDefaultUnitDescription
hdfs_read_buffer_size_kb8192KBSize of the buffer that is used to read data from HDFS.
hdfs_write_buffer_size_kb1024KBSize of the buffer that is used to write data into HDFS.
client_expire_seconds300SecondClient sessions will be deleted if they do not receive any ping after the specified time.
broker_ipc_port8000N/AThe HDFS thrift RPC port.
disable_broker_client_expiration_checkingfalseN/AWhether to disable the checking and clearing of the expired OSS file descriptors, which, in some cases, causes the broker to stuck when OSS is close. To avoid this situation, you can set this parameter to true to disable the checking.
sys_log_dir${BROKER_HOME}/logN/AThe directory used to store system logs (including INFO, WARNING, ERROR, and FATAL).
sys_log_levelINFON/AThe log level. Valid values include INFO, WARNING, ERROR, and FATAL.
sys_log_roll_modeSIZE-MB-1024N/AThe mode how system logs are segmented into log rolls. Valid values include TIME-DAY, TIME-HOUR, and SIZE-MB-nnn. The default value indicates that logs are segmented into rolls which are 1 GB each.
sys_log_roll_num30N/AThe number of log rolls to reserve.
audit_log_dir${BROKER_HOME}/logN/AThe directory that stores audit log files.
audit_log_modulesEmpty stringN/AThe modules for which StarRocks generates audit log entries. By default, StarRocks generates audit logs for the slow_query module and the query module. You can specify multiple modules, whose names must be separated by a comma (,) and a space.
audit_log_roll_modeTIME-DAYN/AValid values include TIME-DAY, TIME-HOUR, and SIZE-MB-<size>.
audit_log_roll_num10N/AThis configuration does not work if the audit_log_roll_mode is set to SIZE-MB-<size>.
sys_log_verbose_modulescom.starrocksN/AThe modules for which StarRocks generates system logs. Valid values are namespaces in BE, including starrocks, starrocks::vectorized, and pipeline.